ChatBots – a small research project

The program (30%) You have been given versions of an ELIZA program (Eliza4 ++), and you should work on this program unless otherwise agreed. ELIZA was the first ChatBot, developed by Joseph Weizenbaum in [login to view URL] aim of the coursework is for you to get some experience, knowledge and understanding of AI+NLP systems. We want you to produce a system that attempts to include some functioning NLP. You don’t have to do it in C++ or start with the code provided, but it gives an excellent starting position and allows the contrast between simple pattern matching and something more intelligent to be brought out. If you don’t use this code, you must agree what you are doing with us, so that we can ensure that what you do fulfils the same criteria. You are required to enhance the code so that the program can carry on a more convincing conversation. Some examples of how this could be done are by developing the program so that it could: bring previous topics into the conversation when relevant; produce grammatically and semantically more complex replies; demonstrate knowledge of topics being talked about; demonstrate learning from the conversation. You should do this by using NLP techniques. Simple versions of NLP techniques will do, but you must explain the technique(s) you are trying to emulate, and how a proper system should work. There are numerous ways that these sorts of things can be done: see Dr. Bowden’s “meta-notes” on NLP and chatbots. One way to approach the coursework is to consider the chatbot working within some application, and the Applications notes might help you there (and see the references below on Intelligent Tutoring Systems (Kerly et al, 2008) and Conversational Agents (Lester et al, 2004)). A very simple example could be being able to ask your chatbot, in natural language, for help with something on the computer – word processing, web searching, finding files, printing, finding things on the internet etc. Or, you might ask it to explain how to solve some mathematical problem, or even to explain some point of grammar/vocab in some human language of your choice. A demonstration of your ChatBot will be required. There are no marks for the demonstration, although you will lose marks if you do not do one. The marks for the program are for your description, explanation and evaluation of it in the report. The demonstration supports your written account. The report (70%) Section 0 - Introduction should explain what chatbots/conversational agents are, and provide some examples of some real chatbots, with brief analyses of their capabilities (3 – 6 pages). 14 marks Section 1 – Description of Starting Code Here you should describe and explain the starting code you have chosen to work with (be it the C++ code originally developed by P. MacDonald, and provided on NOW by us, or another chatbot having code available to you). This means explaining the code and its structure in enough detail for the reader to understand how it works. Don’t forget to describe the purpose and structure of any files that the program reads or writes to. If, instead, you have chosen to develop a chatbot from scratch, then instead of the above you should say why you have decided to do this, and justify your choice of programming language and development environment. (3 - 6 pages). 12 marks Section 2 – Proposed NLP Enhancements Having chosen your application, consider the scope of the application that you aim to achieve (what will it be able to do, and what won’t it be able to do?). Justify and explain the NLP techniques that you propose to use to produce improved (more natural) conversations. Explain how these techniques will work. If you have simplified them for the purposes of this exercise, explain how you have done so and how you would expect them to work in a fully functioning system. Note that this section needs to be written in the future tense – you are stating what you hope to achieve. (about 4 pages). 12 marks Section 3 – Evaluation and Discussion (1) Evaluate your program in detail, so that the reader can decide how close you came to the aims described in the previous section. Explain how you have tested your enhancements and give all relevant results. Illustrate your results with example conversation fragments. (3 – 6 pages) 6 marks (2) Consider how you could improve your program using other NLP techniques, and what a chatbot might need in order to win the Loebner Prize Gold Medal (you will need to describe briefly what the Loebner prize is, of course) (about 3 pages). 6 marks Section 4 - Conclusions Summarise your development and results. Compare the effectiveness of your own chatbot against the real chatbots/conversational agents you discussed in the Introduction of this report. What insights do your and other applications give for the development of NLP technology ? (about 2 pages) 10 marks Overall: correct referencing (Harvard), formatting, presentation, English, spelling etc. 10 marks Submission requirements Your work should be submitted on A4 paper, firmly stapled at the top left hand corner or along the left hand edge. Your work should follow standard reporting requirements unless stated otherwise and should be appropriately word processed. Text must be 12 point Arial, flush left, single spaced. Include your name and student registration number in a header. You must include references (use the Harvard method for referencing); you may include appendices; the main body of the report should be no longer than 22 pages. Do not include a title page, abstract or table of contents; do not use folders or binders of any kind. REPORT The separate sections of the report have marks attached to them, as given above. The mark you actually achieved in each section will reflect how well that section matched the ideal answer, which is a subjective academic judgement by the lecturer. You can assume that 70% or more of the mark for the section represents a first class answer; 60% or more a 2:1; 50% or more a 2:2 and 40% or more a pass. Getting less than 40% of the available marks for a section indicates that you have given an inadequate answer to that section. You will receive a mark for all the sections as detailed above, together with feedback remarks, all written on your reports. To quickly calculate what percentage you received in any section, simply divide the available marks by ten and multiply by 4, 5, 6, 7 etc. For example, if the section had 12 marks available, then 10% of this is 1.2, so 40% is 4.8 i.e. about 5 marks, 50% is 6, 60% is 7.2 i.e. about 7 marks, 70% is 8.4 i.e. about eight and a half marks; 80% is 9.6 etc. The report is marked out of 70 in total, so 49 marks and above is a first class report; 42 – 48 is a 2:1; 35 – 41 is a 2:2; 28 – 34 is a 3rd; less than 28 is a fail. PROGRAM The program is marked out of 30 and reflects the lecturer’s perception of what you have achieved in terms of increased NLP functionality over and above a basic running chatbot. It is therefore vital that your report describes the extra functionality and the code used to deliver it. Clearly this will involve code snippets with explanations, in your report, to describe how you achieved your intended enhancements. The mark out of 30 will be arrived at after considering what you say you have achieved in terms of NLP functionality, the perceived difficulty of what you have attempted (i.e. how ambitious you were), the perceived degree of success in achieving that functionality, the quality of your explanations and evaluation given in the report, and your demonstration. Clearly, a first class program will get 21 marks or over; a 2:1 will get 18 – 20; a 2:2 will get 15 – 17; a 3rd will get 12 – 14; a fail will get 11 or fewer marks. The overall mark (REPORT plus PROGRAM) is out of 70 + 30 = 100 and therefore is a percentage for this assignment (which is half of the entire AI+NLP module marks). About the ELIZA Source Code Provided You have been provided with an ELIZA-type program. This is a small ELIZA shell created by Paul McDonald, written in C++. There’s quite a selection of Eliza programs available over the internet, often written in JAVA, and you could look at these to help develop your ideas. Some examples and useful sources are: [login to view URL] [login to view URL] [login to view URL] and see the ChatBot links; search Google, Yahoo etc for current info.