Specification of Programming #2: Lexical Scanner

1. Read this overview description of the lexical scanner module and play with this zipped sample executable to get a sense of what the lexical scanner module can accomplish for us.

2. References: (i) Review Section 3.2 of C++ Primer (5^th Ed.) to know basics of C++ string class and how to check whether a character is a letter, a digit, or else using library functions in the library <cctype>. (ii) Review Section 3.5.4 about old c style character arrays and how to convert a numeric string (such as “1.23”) stored in a character array into the corresponding numerical value (such as 1.23) using the atof library function in the library <cstdlib>.

3. Download this code framework from the class website as a starting point to get a project of three files: lexScanner.h, lexScanner.cpp, and testScanner.cpp.

• The header file lexScanner.h: It contains information of type definitions of perLineTokenVector, vectOfTokenVects, tokenCategory, perLineCategoryVector, and vectOfCategoryVects and prototypes of the (static) member functions of the LexScanner class that are implemented in lexScanner.cpp and intended to provide service to callers outside lexScanner.cpp. Callers outside lexScanner.cpp can simply include the header file lexScanner.h and then call the static member functions in lexScanner.cpp according to the information in lexScanner.h.
• The source code file lexScanner.cpp: This is the code of the lexical scanner module, and in it we have the implementation of static member functions of the LexScanner class providing lexical analysis service to one another and/or to the outside callers.
• The source code file testScanner.cpp: This is actually not a part of the scanner module. Instead, it simply includes the header file lexScanner.h and is here to play the role of a user of the scanner module to conduct testing of the implementation in lexScanner.cpp.

4. Implementing the LexicalScanner class: Complete the implementation of all the member functions of the LexicalScanner class in lexScanner.cpp

• The header file lexScanner.h: You don’t need to change it. It defines the key types, classes, and data structures needed in this scanner module.
• The source code file lexScanner.cpp: The code in this file is partitioned into 5 sections. The code (member functions) in later sections here often need to call the code (member functions) implemented in earlier sections to finish their job. Explore and understand the overall design of this module first.

i. In Section 1, note that the getPerLineTokenVectFromOneStringObject method is already implemented for you and you can call it to extract all the tokens in a given string object and store the tokens as a vector of string (i.e. a PerLineTokenVect object).

ii. In section 2, there is a collection of member functions for checking the category of a given token. You should fill in code to complete the implementation of several functions as described by the comments to make the code fully functional.

iii. In section 3, you should implement the getCategoryVectorFromTokenVector method, which can retrieve the category information from a given token vector and store them in a category vector.

iv. In section 4, you need to implement the getLexicalInfo method, which is the most important member function to implement in the LexicalScanner class. To implement the getLexicalInfo method, you should use a loop to extract lexical information from the lines of statements one line at a time. The overall task of getLexicalInfo method is to extract from lines of statements and store the lexical information into a vector of token vectors and a vector of category vectors. This is very much like what you did in Lab#2 except that on each iteration (i) you should first call getPerLineTokenVectFromOneStringObject to retrieve the tokens from one line of statement and store them as a token vector, and (ii) you should then call getCategoryVectorFromTokenVector to retrieve the category information from the token vector and store them as a category vector.

v. If you like, you can earn an extra bonus point by writing your own version of getPerLineTokenVectFromOneStringObject function in Section 1 and the displayLexicalInfo function in Section 5 in lexScanner.cpp totally from scratch.

• The source code file testScanner.cpp: While you are implementing functions in lexScanner.cpp, you may want to add extra code here to test whether your implementation works well. The file testScanner.cpp and the code in it is not needed when you later integrate the scanner module with the intereface module as described in Step 5 below.

5. Integration with the interface module

• When you are all set with the implementation in lexScanner.cpp, you should include lexScanner.h and lexScanner.cpp into the project of your user interface module. In your user interface module (interface.cpp), you should implement one more option for the user: the lexical analysis of all lines of statements currently edited by the user. You should implement this by appropriately set up some local data structures (a token vector and a category vector for example) to hold lexical information, passing all lines statements together to the getLexicalInfo member function of the LexicalScanner class to extract and store the lexical information, and then the displayLexicalInfo member function in the LexicalScanner class to display the lexical information. You may look into the main function in testScanner.cpp to see how that could be done.
• If you did not have a working interface module from Programming #1, you can contact me to use my implementation of the interface module.

6. Submission: By the due date, compress and submit your Visual C++ 2103 programming project folder (with source .h & .cpp files in it) and the self-evaluation report of Programming #2 as a zip file under Canvas. Make sure you have done Step 5 to integrate both modules together and add an option of lexical analysis when interacting with the user.

Notes:

• Syntax of BIOLA: our target programming language. Note that the keywords (i.e. function, return, display, read, while, if, else) reserved in this languages are in the red color in the document.
• Expressiveness of the Biola Programming Language: Use the interpreter (download and unzip this zip file) to run this non-trivial sample Biola source program (prime.txt) to see how it can repeatedly find all the prime numbers less than an integer value n given by the user.