Upgraded database,

Added database documentation

docs/dev/db_infos.md

@@ -0,0 +1,94 @@
+# Database Documentation
+
+## Introduction
+This project implements a discussion forum and utilizes a SQLite database to store discussions and messages.
+
+## Database Schema
+The following database schema has been implemented:
+
+- `discussion`: table to store discussion information including id and title.
+- `message`: table to store message information including id, sender, content, type, rank, parent, and discussion_id. The type column is an integer representing the type of the message. The rank column is an integer representing the rank of the message. The parent column is an integer representing the id of the parent message. The discussion_id column is a foreign key referencing the id column in the discussion table.
+
+## Database Upgrades
+The database schema is currently at version 2. The following upgrades have been made to the schema:
+
+In version 1, three columns have been added to the message table: type, rank, and parent.
+In version 2, the parent column has been added to the message table (if it doesn't already exist).
+Encoding
+The encoding of the database is checked before creating/updating the schema. If the current encoding is not UTF-8, it is changed to UTF-8.
+
+## Implementation Details
+The create database schema script is responsible for creating/updating the database schema. It first checks the encoding of the database and changes it to UTF-8 if necessary. Then, it checks if the required tables (discussion, message, and schema_version) exist. If any of these tables do not exist, they are created. The schema version is retrieved from the schema_version table. If the table is empty, version 0 is assumed. Otherwise, the version from the table is used. If the version is less than the current version, the schema is upgraded to the current version by adding the necessary columns to the message table. Finally, the schema version is updated or inserted into the schema_version table.
+
+## Documentation:
+
+The DiscussionDB class provides methods to manipulate the database and retrieve information about discussions and messages. Here are the methods available in this class:
+```
+select(query, params=None, fetch_all=True)
+```
+
+This method executes an SQL select query on the database with optional parameters. It returns the cursor object for further processing. If fetch_all is True, it returns all the rows returned by the query, else it returns only the first row.
+
+```
+delete(query, params=None)
+```
+
+This method executes an SQL delete query on the database with optional parameters. It returns the cursor object for further processing.
+
+```
+insert(query, params=None)
+```
+
+This method executes an SQL insert query on the database with optional parameters. It returns the ID of the newly inserted row.
+
+```
+update(query, params=None)
+```
+
+This method executes an SQL update query on the database with optional parameters.
+
+```
+load_last_discussion()
+```
+
+This method retrieves the last discussion in the database or creates a new one if there are no discussions. It returns a Discussion instance for the retrieved or created discussion.
+
+```
+create_discussion(title="untitled")
+```
+
+This method creates a new discussion with the specified title. It returns a Discussion instance for the newly created discussion.
+
+```
+build_discussion(discussion_id=0)
+```
+
+This method retrieves the discussion with the specified ID or creates a new one if the ID is not found. It returns a Discussion instance for the retrieved or created discussion.
+
+```
+get_discussions()
+```
+
+This method retrieves all discussions in the database. It returns a list of dictionaries, where each dictionary represents a discussion and contains the discussion's ID and title.
+
+```
+does_last_discussion_have_messages()
+```
+
+This method checks if the last discussion in the database has any messages. It returns True if the last discussion has at least one message, else False.
+
+```
+remove_discussions()
+```
+
+This method removes all discussions and messages from the database.
+
+```
+export_to_json()
+```
+
+This method exports all discussions and messages in the database to a JSON file. It returns a list of dictionaries, where each dictionary represents a discussion and contains the discussion's ID, title, and messages. Each message is represented by a dictionary with the sender, content, type, rank, and parent fields.
+
+
+## Database diagram
+[](scheme.png)

personalities/english/general/gpt4all/config.yaml

@@ -7,7 +7,7 @@
 # talking to.
 
 #The version of the PyAIPersonality used to build this file
-pyaipersonality_version: 0.0.2
+pyaipersonality_version: 0.0.5
 
 #The version of the personality
 version: 1.0.0
@@ -30,24 +30,44 @@ personality_description: |
 
 # The conditionning instructions sent to eh model at the start of the discussion
 personality_conditioning: |
-  ##Instructions:\nGPT4All is a smart and helpful Assistant built by Nomic-AI. It can discuss with humans and assist them.
+  ## Information:
+  Assistant's name is gpt4all
+  Today's date is {{date}}
+  ## Instructions:
+  Your mission is to assist user to perform various tasks and answer his questions
 
 #Welcome message to be sent to the user when a new discussion is started
-welcome_message: "Welcome! I am GPT4All A free and open assistant. What can I do for you today?"
+welcome_message: |
+                    Welcome! My name is gpt4all.
+                    How can I help you today?
 
 # This prefix is added at the beginning of any message input by the user
-user_message_prefix:  "##Human:\n"
-
+user_message_prefix:  "###user:
+                      
+                      "
 # A text to put between user and chatbot messages
 link_text: "\n"
 
 # This prefix is added at the beginning of any message output by the ai
-ai_message_prefix: "##Assistant:\n"
+ai_message_prefix: "###gpt4all:
+                   
+                   "
 
 # Here is the list of extensions this personality requires
 dependencies: []
 
+# A list of texts to be used to detect that the model is hallucinating and stop the generation if any one of these is output by the model
+anti_prompts: ["###user","### user","###gpt4all","### gpt4all"]
+
 # Some personalities need a disclaimer to warn the user of potential harm that can be caused by the AI
 # for example, for medical assistants, it is important to tell the user to be careful and not use medication
 # without advise from a real docor.
-disclaimer: ""
+disclaimer: ""
+
+# Here are default model parameters
+model_temperature: 0.6 # higher: more creative, lower more deterministic
+model_n_predicts: 1024 # higher: generates many words, lower generates
+model_top_k: 50
+model_top_p: 0.90
+model_repeat_penalty: 1.0
+model_repeat_last_n: 40

pyGpt4All/api.py

@@ -177,7 +177,7 @@ class GPT4AllAPI():
         sys.stdout.flush()
         
         self.bot_says += text
-        if not self.personality.user_message_prefix.strip().lower() in self.bot_says.lower():
+        if not self.personality.detect_antiprompt(self.bot_says):
             self.socketio.emit('message', {'data': self.bot_says});
             if self.cancel_gen:
                 print("Generation canceled")

pyGpt4All/db.py

@@ -19,7 +19,7 @@ class DiscussionsDB:
         """
         create database schema
         """
-        db_version = 2
+        db_version = 3
         # Verify encoding and change it if it is not complient
         with sqlite3.connect(self.db_path) as conn:
             # Execute a PRAGMA statement to get the current encoding of the database
@@ -52,10 +52,11 @@ class DiscussionsDB:
                 schema_table_exist = True
             try:
                 cursor.execute("""
-                        CREATE TABLE discussion (
-                            id INTEGER PRIMARY KEY AUTOINCREMENT,
-                            title TEXT
-                        )
+                    CREATE TABLE discussion (
+                        id INTEGER PRIMARY KEY AUTOINCREMENT,
+                        title TEXT,
+                        created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
+                    )
                     """)
             except Exception:
                 discussion_table_exist=True        
@@ -68,8 +69,10 @@ class DiscussionsDB:
                             type INT NOT NULL,
                             rank INT NOT NULL,
                             parent INT,
+                            created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
                             discussion_id INTEGER NOT NULL,
-                            FOREIGN KEY (discussion_id) REFERENCES discussion(id)
+                            FOREIGN KEY (discussion_id) REFERENCES discussion(id),
+                            FOREIGN KEY (parent) REFERENCES message(id)
                         )
                     """
                     )
@@ -94,6 +97,9 @@ class DiscussionsDB:
                     cursor.execute("ALTER TABLE message ADD COLUMN type INT DEFAULT 0") # Added in V1
                     cursor.execute("ALTER TABLE message ADD COLUMN rank INT DEFAULT 0") # Added in V1
                     cursor.execute("ALTER TABLE message ADD COLUMN parent INT DEFAULT 0") # Added in V2
+                    cursor.execute("ALTER TABLE message ADD COLUMN created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP") # Added in V3
+                    cursor.execute("ALTER TABLE discussion ADD COLUMN created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP") # Added in V3
+                    
             # Upgrade the schema to version 1
             elif version < 2:
                 print(f"Upgrading schema to version {db_version}...")
@@ -101,6 +107,18 @@ class DiscussionsDB:
                 if message_table_exist:
                     try:
                         cursor.execute("ALTER TABLE message ADD COLUMN parent INT DEFAULT 0") # Added in V2
+                        cursor.execute("ALTER TABLE message ADD COLUMN created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP") # Added in V3
+                        cursor.execute("ALTER TABLE discussion ADD COLUMN created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP") # Added in V3
+                    except :
+                        pass
+            # Upgrade the schema to version 1
+            elif version < 3:
+                print(f"Upgrading schema to version {db_version}...")
+                # Add the 'created_at' column to the 'message' table
+                if message_table_exist:
+                    try:
+                        cursor.execute("ALTER TABLE message ADD COLUMN created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP") # Added in V3
+                        cursor.execute("ALTER TABLE discussion ADD COLUMN created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP") # Added in V3
                     except :
                         pass
             # Update the schema version

requirements.txt

@@ -14,4 +14,4 @@ transformers
 accelerate
 gevent
 gevent-websocket
-pyaipersonality
+pyaipersonality==0.0.5