LLM/khoj
1
0
Fork 0
mirror of https://github.com/khoj-ai/khoj.git synced 2026-05-13 21:41:41 +00:00
Code Issues Packages Projects Releases Wiki Activity

Compare commits

base: LLM:0.2.5
Branches Tags
LLM:master LLM:release/1.x LLM:v2/add-live-voice-chat LLM:debanjum/miscellaneous-v2-changes LLM:use-user-specified-chat-model-for-intermediate-steps LLM:pr/hjamet/1109 LLM:create-data-tracer LLM:evals/tmp-distill-trains-of-thought LLM:load-web-app-from-desktop-app LLM:use-structured-index-to-improve-kb-retrieval LLM:speed-up-khoj-docker-builds LLM:features/support-sqlite-as-db LLM:regenerate-edit-messages-on-web-app LLM:improve-agents-pane-on-home-screen LLM:use-turnstyle-ux-for-khoj-chat LLM:improve-doc-chunking-and-support-timely-embedings LLM:speed-up-embedding-model-app-load-when-not-connected-to-internet LLM:panchayat
LLM:2.0.0-beta.28 LLM:2.0.0-beta.27 LLM:2.0.0-beta.26 LLM:2.0.0-beta.25 LLM:2.0.0-beta.24 LLM:2.0.0-beta.23 LLM:2.0.0-beta.22 LLM:2.0.0-beta.21 LLM:2.0.0-beta.20 LLM:2.0.0-beta.19 LLM:2.0.0-beta.18 LLM:2.0.0-beta.17 LLM:2.0.0-beta.16 LLM:2.0.0-beta.15 LLM:2.0.0-beta.14 LLM:2.0.0-beta.13 LLM:2.0.0-beta.12 LLM:2.0.0-beta.11 LLM:2.0.0-beta.10 LLM:2.0.0-beta.9 LLM:2.0.0-beta.8 LLM:1.42.10 LLM:2.0.0-beta.7 LLM:2.0.0-beta.6 LLM:2.0.0-beta.5 LLM:2.0.0-beta.4 LLM:2.0.0-beta.3 LLM:2.0.0-beta.2 LLM:2.0.0-beta.1 LLM:1.42.9 LLM:1.42.8 LLM:1.42.7 LLM:1.42.6 LLM:1.42.5 LLM:1.42.4 LLM:1.42.3 LLM:1.42.2 LLM:1.42.1 LLM:1.42.0 LLM:1.41.0 LLM:1.40.0 LLM:1.39.0 LLM:1.38.0 LLM:1.37.2 LLM:1.37.1 LLM:1.37.0 LLM:1.36.6 LLM:1.36.5 LLM:1.36.4 LLM:1.36.3 LLM:1.36.2 LLM:1.36.1 LLM:1.36.0 LLM:1.35.3 LLM:1.35.2 LLM:1.35.1 LLM:1.35.0 LLM:1.34.0 LLM:1.33.2 LLM:1.33.1 LLM:1.33.0 LLM:1.32.2 LLM:1.32.1 LLM:1.32.0 LLM:1.31.0 LLM:1.30.10 LLM:1.30.9 LLM:1.30.8 LLM:1.30.7 LLM:1.30.6 LLM:1.30.5 LLM:1.30.4 LLM:1.30.3 LLM:1.30.2 LLM:1.30.1 LLM:1.30.0 LLM:1.29.1 LLM:1.29.0 LLM:1.28.3 LLM:1.28.2 LLM:1.28.1 LLM:1.28.0 LLM:1.27.1 LLM:1.27.0 LLM:1.26.4 LLM:1.26.3 LLM:1.26.2 LLM:1.26.1 LLM:1.26.0 LLM:1.25.0 LLM:1.24.1 LLM:1.24.0 LLM:1.23.3 LLM:1.23.2 LLM:1.23.1 LLM:1.23.0 LLM:1.22.3 LLM:1.22.2 LLM:1.22.1 LLM:1.22.0 LLM:1.21.6 LLM:1.21.5 LLM:1.21.4 LLM:1.21.3 LLM:1.21.2 LLM:1.21.1 LLM:1.21.0 LLM:1.20.4 LLM:1.20.3 LLM:1.20.2 LLM:1.20.1 LLM:1.20.0 LLM:1.17.0 LLM:1.16.0 LLM:1.15.0 LLM:1.14.0 LLM:1.13.0 LLM:1.12.1 LLM:1.12.0 LLM:1.11.2 LLM:1.11.1 LLM:1.11.0 LLM:1.10.2 LLM:1.10.1 LLM:1.10.0 LLM:1.9.0 LLM:1.8.0 LLM:1.7.0 LLM:1.6.2 LLM:1.6.1 LLM:1.6.0 LLM:1.5.1 LLM:1.5.0 LLM:1.4.0 LLM:1.3.0 LLM:1.2.1 LLM:1.2.0 LLM:1.1.0 LLM:1.0.1 LLM:1.0.0 LLM:0.14.0 LLM:0.13.0 LLM:0.12.3 LLM:0.12.2 LLM:0.12.1 LLM:rm LLM:0.12.0 LLM:0.11.4 LLM:0.11.3 LLM:0.11.2 LLM:0.11.1 LLM:0.11.0 LLM:0.10.1 LLM:0.10.0 LLM:0.9.0 LLM:0.8.2 LLM:0.8.1 LLM:0.8.0 LLM:0.7.1 LLM:0.7.0 LLM:0.6.2 LLM:0.6.1 LLM:0.6.0 LLM:0.5.0 LLM:0.4.0 LLM:0.3.0 LLM:0.2.6 LLM:0.2.5 LLM:0.2.1 LLM:v0.2.0 LLM:v0.1.9 LLM:v0.1.8 LLM:v0.1.7 LLM:v0.1.6 LLM:v0.1.5-alpha
...
compare: LLM:0.8.0
Branches Tags
LLM:master LLM:release/1.x LLM:v2/add-live-voice-chat LLM:debanjum/miscellaneous-v2-changes LLM:use-user-specified-chat-model-for-intermediate-steps LLM:pr/hjamet/1109 LLM:create-data-tracer LLM:evals/tmp-distill-trains-of-thought LLM:load-web-app-from-desktop-app LLM:use-structured-index-to-improve-kb-retrieval LLM:speed-up-khoj-docker-builds LLM:features/support-sqlite-as-db LLM:regenerate-edit-messages-on-web-app LLM:improve-agents-pane-on-home-screen LLM:use-turnstyle-ux-for-khoj-chat LLM:improve-doc-chunking-and-support-timely-embedings LLM:speed-up-embedding-model-app-load-when-not-connected-to-internet LLM:panchayat
LLM:2.0.0-beta.28 LLM:2.0.0-beta.27 LLM:2.0.0-beta.26 LLM:2.0.0-beta.25 LLM:2.0.0-beta.24 LLM:2.0.0-beta.23 LLM:2.0.0-beta.22 LLM:2.0.0-beta.21 LLM:2.0.0-beta.20 LLM:2.0.0-beta.19 LLM:2.0.0-beta.18 LLM:2.0.0-beta.17 LLM:2.0.0-beta.16 LLM:2.0.0-beta.15 LLM:2.0.0-beta.14 LLM:2.0.0-beta.13 LLM:2.0.0-beta.12 LLM:2.0.0-beta.11 LLM:2.0.0-beta.10 LLM:2.0.0-beta.9 LLM:2.0.0-beta.8 LLM:1.42.10 LLM:2.0.0-beta.7 LLM:2.0.0-beta.6 LLM:2.0.0-beta.5 LLM:2.0.0-beta.4 LLM:2.0.0-beta.3 LLM:2.0.0-beta.2 LLM:2.0.0-beta.1 LLM:1.42.9 LLM:1.42.8 LLM:1.42.7 LLM:1.42.6 LLM:1.42.5 LLM:1.42.4 LLM:1.42.3 LLM:1.42.2 LLM:1.42.1 LLM:1.42.0 LLM:1.41.0 LLM:1.40.0 LLM:1.39.0 LLM:1.38.0 LLM:1.37.2 LLM:1.37.1 LLM:1.37.0 LLM:1.36.6 LLM:1.36.5 LLM:1.36.4 LLM:1.36.3 LLM:1.36.2 LLM:1.36.1 LLM:1.36.0 LLM:1.35.3 LLM:1.35.2 LLM:1.35.1 LLM:1.35.0 LLM:1.34.0 LLM:1.33.2 LLM:1.33.1 LLM:1.33.0 LLM:1.32.2 LLM:1.32.1 LLM:1.32.0 LLM:1.31.0 LLM:1.30.10 LLM:1.30.9 LLM:1.30.8 LLM:1.30.7 LLM:1.30.6 LLM:1.30.5 LLM:1.30.4 LLM:1.30.3 LLM:1.30.2 LLM:1.30.1 LLM:1.30.0 LLM:1.29.1 LLM:1.29.0 LLM:1.28.3 LLM:1.28.2 LLM:1.28.1 LLM:1.28.0 LLM:1.27.1 LLM:1.27.0 LLM:1.26.4 LLM:1.26.3 LLM:1.26.2 LLM:1.26.1 LLM:1.26.0 LLM:1.25.0 LLM:1.24.1 LLM:1.24.0 LLM:1.23.3 LLM:1.23.2 LLM:1.23.1 LLM:1.23.0 LLM:1.22.3 LLM:1.22.2 LLM:1.22.1 LLM:1.22.0 LLM:1.21.6 LLM:1.21.5 LLM:1.21.4 LLM:1.21.3 LLM:1.21.2 LLM:1.21.1 LLM:1.21.0 LLM:1.20.4 LLM:1.20.3 LLM:1.20.2 LLM:1.20.1 LLM:1.20.0 LLM:1.17.0 LLM:1.16.0 LLM:1.15.0 LLM:1.14.0 LLM:1.13.0 LLM:1.12.1 LLM:1.12.0 LLM:1.11.2 LLM:1.11.1 LLM:1.11.0 LLM:1.10.2 LLM:1.10.1 LLM:1.10.0 LLM:1.9.0 LLM:1.8.0 LLM:1.7.0 LLM:1.6.2 LLM:1.6.1 LLM:1.6.0 LLM:1.5.1 LLM:1.5.0 LLM:1.4.0 LLM:1.3.0 LLM:1.2.1 LLM:1.2.0 LLM:1.1.0 LLM:1.0.1 LLM:1.0.0 LLM:0.14.0 LLM:0.13.0 LLM:0.12.3 LLM:0.12.2 LLM:0.12.1 LLM:rm LLM:0.12.0 LLM:0.11.4 LLM:0.11.3 LLM:0.11.2 LLM:0.11.1 LLM:0.11.0 LLM:0.10.1 LLM:0.10.0 LLM:0.9.0 LLM:0.8.2 LLM:0.8.1 LLM:0.8.0 LLM:0.7.1 LLM:0.7.0 LLM:0.6.2 LLM:0.6.1 LLM:0.6.0 LLM:0.5.0 LLM:0.4.0 LLM:0.3.0 LLM:0.2.6 LLM:0.2.5 LLM:0.2.1 LLM:v0.2.0 LLM:v0.1.9 LLM:v0.1.8 LLM:v0.1.7 LLM:v0.1.6 LLM:v0.1.5-alpha

498 Commits
0.2.5 ... 0.8.0

Author SHA1 Message Date
Debanjum Singh Solanky
75ff871217 Release Khoj version 0.8.0 2023-07-10 13:37:51 -07:00
Debanjum Singh Solanky
979088b3dc Add tooltip helper text on web settings page buttons 
- Provide more details on what clicking configure, initialize buttons
  or changing the results count slider does
- This shows up on user hovering over those buttons
 2023-07-10 13:32:41 -07:00
Debanjum Singh Solanky
255781e135 Use relative link on logo to jump to correct page on local and cloud 2023-07-10 13:22:20 -07:00
Debanjum Singh Solanky
b2d229c116 Move header pane style to base khoj.css for reuse. Fix logo size 2023-07-10 13:10:17 -07:00
Debanjum Singh Solanky
f4cef377ca Add details to run, configure Khoj via Web in Readme 2023-07-10 12:10:20 -07:00
Debanjum Singh Solanky
20cb314171 Open the Khoj config page in the browser on first run 2023-07-10 12:10:20 -07:00
sabaimran
07cf5a214a Check if PDF files are present in the Obsidian vault before initializing the Khoj configuration (#293) 2023-07-10 10:33:04 -07:00
sabaimran
7364bac8ae Make the header take up less space 
- Use a single row for the header
- Needed custom styling for each page because each of them are different in subtle ways, unfortunately
 2023-07-09 22:31:37 -07:00
sabaimran
62704cac09 Add a plugin which allows users to index their Notion pages (#284) 
* For the demo instance, re-instate the scheduler, but infrequently for api updates
- In constants, determine the cadence based on whether it's a demo instance or not
- This allow us to collect telemetry again. This will also allow us to save the chat session
* Conditionally skip updating the index altogether if it's a demo isntance
* Add backend support for Notion data parsing
- Add a NotionToJsonl class which parses the text of Notion documents made accessible to the API token
- Make corresponding updates to the default config, raw config to support the new notion addition
* Add corresponding views to support configuring Notion from the web-based settings page
- Support backend APIs for deleting/configuring notion setup as well
- Streamline some of the index updating code
* Use defaults for search and chat queries results count
* Update pagination of retrieving pages from Notion
* Update state conversation processor when update is hit
* frequency_penalty should be passed to gpt through kwargs
* Add check for notion in render_multiple method
* Add headings to Notion render
* Revert results count slider and split Notion files by blocks
* Clean/fix misc things in the function to update index
- Use the successText and errorText variables appropriately
- Name parameters in function calls
- Add emojis, woohoo
* Clean up and further modularize code for processing data in Notion
 2023-07-09 15:29:26 -07:00
Debanjum
77755c0284 Fix Packaging the Khoj Desktop Apps (#289) 
* Add langchain static files and pytorch metadata to Khoj native app

* Add pillow static files, metadata & hidden imports to Khoj native app

* Fix path to web interface static files on Khoj native app

* Add tiktoken hidden imports to make chat work from Khoj native app

* Fix Khoj native app to run with GUI mode enabled

This got broken when we moved from using the --no-gui flag to using
--gui in https://github.com/khoj-ai/khoj/pull/263
 2023-07-09 10:21:16 -07:00
sabaimran
4c135ea316 Make streaming optional for the /chat endpoint (#287) 
* Update the /chat endpoint to conditionally support streaming

- If streams are enabled, return the threadgenerator as it does currently
- If stream is disabled, return a JSON response with the response/compiled references separated out
- Correspondingly, update the chat.html UI to use the streamed API, as well as Obsidian
- Rename chat/init/ to chat/history

* Update khoj.el to use the /history endpoint

- Update corresponding unit tests to use stream=true

* Remove & from call to /chat for obsidian

* Abstract functions out into a helpers.py file and clean up some of the error-catching
 2023-07-09 10:12:09 -07:00
Debanjum Singh Solanky
0a86220d42 Use default values, delete content config on disable and update state 2023-07-07 20:36:16 -07:00
Debanjum Singh Solanky
362063f5fe By default, connect to Khoj server over IPv4 from Obsidian plugin 2023-07-07 20:36:16 -07:00
Debanjum Singh Solanky
571e8c2548 Add rerank, index corruption hint on search page of web interface 
Similar to the hint alrady in the Obsidian search modal
Closes #272
 2023-07-07 20:36:16 -07:00
Debanjum
4b79d8216f Move remaining chat actors to use OpenAI chat models 
- Deprecate the unused beta /answer and /search type identification endpoints and associated GPT functions
- Update extract_questions to use GPT4
- Update summarize method to default to GPT-3.5
- Update date filter to support quoting values in single quotes too. So now both dt>'2023-04-01' and dt>"2023-04-01" should work
- Remove "model" field from chat settings on the web interface
 2023-07-07 18:53:05 -07:00
Debanjum Singh Solanky
61e131f95c Hide unused model field from chat settings on web interface 2023-07-07 18:43:53 -07:00
Debanjum Singh Solanky
af30d01e85 Move to newer chat models to extract questions & summarize chats 
Deprecate usage of the older gpt3 models in-place of the newer chat
based models
- text-davinci-003 is only 50% cheaper than gpt4 and less reliable for
  question extraction
- Using gpt-3.50turbo for summarization should reduce cost of chat

- Keep conversation.chat_session as a list instead of a string
- Update completion_with_backoff func to use ChatML format
 2023-07-07 17:32:27 -07:00
Debanjum Singh Solanky
171ce19e1f Update date filter to allow quoting values in single quotes 2023-07-07 17:13:47 -07:00
Debanjum Singh Solanky
e588f7c528 Deprecate unused beta search and answer API endpoints 2023-07-07 16:38:07 -07:00
Debanjum Singh Solanky
c9fc4d1296 Revert to using cross-encoder to improve search results used by chat 2023-07-07 15:31:34 -07:00
Debanjum Singh Solanky
11f0a9f196 Fix chat tests since streaming. Pass args correctly to chat methods 
- Fix testing gpt converse method after it started streaming responses
- Pass stop in model_kwargs dictionary and api key in openai_api_key
  parameter to chat completion methods. This should resolve the arg
  warning thrown by OpenAI module
 2023-07-07 15:23:44 -07:00
Debanjum Singh Solanky
48870d9170 Fix parsing questions generated by extract_questions actor into list 
The previous json parsing was failing to handle questions with date
filters

Fix the chat actor tests to run without throwing error with freezegun
complaining about importing transformers.local_llama model

Remove quote escapes from date filter examples provided to
extract_questions actor
 2023-07-07 15:18:55 -07:00
Debanjum Singh Solanky
279662620b Move results count to settings page on web. Use it for search & chat 
- Before
  Only the search interface had the results count configuration option

- After
  - The results count is set on the settings page instead of the
    search page
  - Both search and chat can use the configured results count instead
    of just search
 2023-07-07 14:08:08 -07:00
Debanjum Singh Solanky
2ec8da89e8 Remove Update button from Khoj Search page on the Web interface 
The settings page on the Khoj web interface already has a configure
button. Don't need the Update button on the search page as well
 2023-07-07 12:49:58 -07:00
Debanjum Singh Solanky
bf427cd8dd Set no. of results used to generate chat response from Khoj Emacs 2023-07-07 12:34:50 -07:00
Debanjum Singh Solanky
1d77fe712c Set no. of results used to generate chat response from Khoj Obsidian 2023-07-07 12:32:32 -07:00
Debanjum Singh Solanky
2f31de5ed5 Set no. of references to use for chat configurable in Chat API 2023-07-07 12:29:36 -07:00
Debanjum Singh Solanky
d97682fdac Use tooltip, placeholders to guide Khoj setup via web settings page 2023-07-06 21:37:48 -07:00
Debanjum Singh Solanky
f5cf09424b Use more descriptive field names for content type settings on Khoj web 
Resolves #281
 2023-07-06 20:47:39 -07:00
Debanjum Singh Solanky
a2c668268f Use node-fetch >=3.1.0 in khoj obsidian plugin to avoid security vulnerability 2023-07-06 13:05:52 -07:00
sabaimran
d688ddf92c Re-instate the scheduler for the demo instances (#279) 
* For the demo instance, re-instate the scheduler, but infrequently for api updates

- In constants, determine the cadence based on whether it's a demo instance or not
- This allow us to collect telemetry again. This will also allow us to save the chat session

* Conditionally skip updating the index altogether if it's a demo isntance
 2023-07-06 11:01:32 -07:00
Debanjum Singh Solanky
8f36572a9b Improve typing, null checks in controllers and gpt functions 2023-07-05 20:49:25 -07:00
Debanjum Singh Solanky
41ac1e24c9 Add docs for a pre-emptive setup of Khoj for later offline usage 
Closes #151
 2023-07-05 20:48:51 -07:00
Debanjum
6c2a8a5bce ⚡️ Stream Responses by Khoj Chat on Web, Obsidian 
- What
   - Stream chat responses from OpenAI API to Web, Obsidian clients
      - Implement using a callback function which manages a queue where new tokens can be placed as they come on. As the thread is read from, tokens are removed.
      - When the final token has been processed, add the `compiled_references` to the queue to be rendered by the `chat` client
      - When the thread has been closed, save the accumulated conversation log in the user's history using a `partial func`
      - Incrementally decode tokens on the front end and add them as they appear from the streamed response

- Why
This significantly reduces perceived latency and OpenAI API request timeouts for Chat

Closes https://github.com/khoj-ai/khoj/issues/257
 2023-07-05 20:02:11 -07:00
Debanjum Singh Solanky
e111eda6ae Make client, app_config optional in telemetry logger for correct typing 2023-07-05 18:57:38 -07:00
Debanjum Singh Solanky
e562114f6b Improve comments, var names in js for chat streaming on web interface 2023-07-05 18:57:27 -07:00
Debanjum Singh Solanky
46269ddfd3 Fix chat logging messages to get context without flooding logs 2023-07-05 18:27:06 -07:00
Debanjum Singh Solanky
0ba838b53a Show temp status message in Khoj Obsidian chat while Khoj is thinking 
- Scroll to bottom after adding temporary status message and
references too
 2023-07-05 18:02:43 -07:00
Debanjum Singh Solanky
8271abe729 Use optional chaining operator to extract khojBannerSubmit from conditional 2023-07-05 18:02:43 -07:00
Debanjum Singh Solanky
c12ec1fd03 Show temp status message in Khoj web chat while Khoj is thinking 
- Scroll to bottom after adding temporary status message and
references too
 2023-07-05 18:02:30 -07:00
sabaimran
257a421e45 Bonus: add try-catch logic around telemetry upload in case of JSON serializability issues 2023-07-05 15:12:18 -07:00
sabaimran
4e6b66b139 Add support for streaming chat response from OpenAI to Obsidian 
- I needed to installed node-fetch to accomplish this, as the built-in request object from Obsidian doesn't seem to support streaming and the built-in fetch object is very sensitive to any and all cross origin requests
 2023-07-05 15:01:22 -07:00
sabaimran
3ff5074cf5 Log the end-to-end time of generating a streamed response from OpenAI 2023-07-05 14:59:44 -07:00
sabaimran
68e635cc32 Remove additional comments and debug statements 2023-07-05 11:33:56 -07:00
sabaimran
67a8795b1f Clean-up commented out code 2023-07-05 11:24:40 -07:00
sabaimran
79b1b1d350 Save streamed chat conversations via partial function passed to the ThreadGenerator 2023-07-04 17:33:52 -07:00
sabaimran
afd162de01 Add reference notes to result response from GPT when streaming is completed 
- NOTE: results are still not being saved to conversation history
 2023-07-04 12:47:50 -07:00
sabaimran
8f491d72de Initial code with chat streaming working (warning: messy code) 2023-07-04 10:14:39 -07:00
Debanjum Singh Solanky
5889eceba4 Make text selectable in Khoj chat modal on Obsidian 
Previously the text in the Khoj chat modal couldn't be copied as it
was not selectable

Resolves #206
 2023-07-03 23:24:04 -07:00
sabaimran
89354def9b Update request timeout window to 20 seconds 2023-07-03 22:28:18 -07:00
sabaimran
b1940519c3 Log error if unable to decode chunk from Github 2023-07-03 16:29:32 -07:00
Debanjum Singh Solanky
ecf9730cd7 Disable Chat, Search on Web if Khoj not configured & show next steps 2023-07-03 16:04:32 -07:00
sabaimran
017e8c1aef Skip indexing a PDF that has an indexing error (#274) 2023-07-03 15:55:11 -07:00
sabaimran
a6f313589e Release Khoj version 0.7.1 2023-07-03 12:26:41 -07:00
Debanjum Singh Solanky
70f6b8266c Upgrade minimum supported pydantic version 2023-07-03 12:22:56 -07:00
sabaimran
8bfd5828e6 Remove deprecation notice since we're opening the web UI by default 2023-07-03 12:01:09 -07:00
sabaimran
92d81d3b16 Initialize the search.model field to SearchModels() and fix Reinitialize API call (#273) 2023-07-03 11:32:44 -07:00
sabaimran
61403138d5 Merge pull request #269 from khoj-ai/features/simplify-configuration-steps 
Simplify some common configuration steps
 2023-07-03 00:16:51 -07:00
sabaimran
ea3dc2cfa3 Simplify rendering of content type pages and logic of selecting config 2023-07-03 00:15:29 -07:00
sabaimran
260272dca2 Check if state.config is populated before configuring via the update method 2023-07-03 00:10:56 -07:00
sabaimran
bf8914d0c8 Fix default config initialization for for chat.html 2023-07-03 00:00:47 -07:00
Debanjum
faad1297f4 Drop Support for Org Music, Ledger Content Types 
Removing unused content types will reduce khoj code to manage

- 0f993b3 Drop support for Ledger as a separate content type
   Khoj will soon get a generic text indexing content type in Index plain text files #237.
   This along with a file filter should suffice for searching through Ledger transactions

- c9db532 Remove unused org-music as an indexable content type from Khoj
   Org-music was just a custom content type that worked with org-music.
   It was mostly only useful for me.
 2023-07-02 17:48:29 -07:00
Debanjum Singh Solanky
0f993b332e Drop support for Ledger as a separate content type 
Khoj will soon get a generic text indexing content type. This along
with a file filter should suffice for searching through Ledger
transactions, if required.

Having a specific content type for niche use-case like ledger isn't
useful. Removing unused content types will reduce khoj code to manage.
 2023-07-02 16:57:49 -07:00
sabaimran
fa218ff5aa Fix call to update for Reinitialize button 2023-07-02 16:31:30 -07:00
sabaimran
a8b83da872 Merge branch 'master' of github.com:debanjum/khoj into features/simplify-configuration-steps 2023-07-02 16:21:54 -07:00
Debanjum Singh Solanky
c9db5321e7 Remove unused org-music as an indexable content type from Khoj 
Org-music was just a custom content type that worked with org-music.
It was mostly only useful for me.

Cleaning up that code will reduce number of content types for khoj to
manage.
 2023-07-02 16:21:21 -07:00
sabaimran
77a45f4215 Merge pull request #265 from khoj-ai/fix/obsidian-setup-issues 
Fix configuration setup logic in Obsidian
 2023-07-02 16:21:18 -07:00
sabaimran
b86a3bb0c5 Merge branch 'master' of github.com:debanjum/khoj into fix/obsidian-setup-issues 2023-07-02 16:21:05 -07:00
sabaimran
a52c1c8380 Use built-in app.vault to determine whether there are any PDF files within 2023-07-02 16:20:43 -07:00
sabaimran
eff1436857 Overwrite existing PDFs in Obsidian as well, make if-block more legible 2023-07-02 16:17:25 -07:00
Debanjum Singh Solanky
30459ee4ba Fix Khoj subtitle in desktop entry, pyproject, cli and Obsidian Readme 2023-07-02 16:09:07 -07:00
sabaimran
feac71ce1e Merge pull request #268 from khoj-ai/fix/threading-issue-in-update-api 
Add try-except-finally blocks around configure calls in /update
 2023-07-02 16:08:29 -07:00
sabaimran
1a1b044d12 Simplify settings pages for configuration 
- Add one-click disablement
- Remove fields that probably don't need to be edited (our implementation details)
- Add a green tick if a given field is configured
 2023-07-02 16:04:05 -07:00
sabaimran
e4c445f805 Add try-except-finally blocks around configure calls in /update 2023-07-02 13:35:02 -07:00
sabaimran
4b02a8c788 Fix PDF setup in Obsidian plugin and force Obsidian configuration for markdown 2023-07-02 12:37:24 -07:00
sabaimran
b6772d8fc3 Merge pull request #264 from khoj-ai/fix/remove-guidance-for-desktop-gui 
Escape special characters in the URL when adding a link to the remote file
 2023-07-02 09:14:08 -07:00
sabaimran
2a7e4f2b71 Escape special characters in the URL when adding a link to the remote file 2023-07-02 09:13:28 -07:00
sabaimran
4915b7214d Merge pull request #263 from khoj-ai/fix/remove-guidance-for-desktop-gui 
[Fix] Remove the default behavior of using GUI for Khoj
 2023-07-01 21:37:11 -07:00
sabaimran
c747562897 Update the GUI to just be a simple box with a button for the web UI 2023-07-01 20:37:21 -07:00
sabaimran
bab7f39d47 Move logic to open the web browser into the GUI section 2023-07-01 20:11:27 -07:00
sabaimran
36537606da Update unit test and preserve prior operational ordering in main.py 2023-07-01 20:02:35 -07:00
sabaimran
ea9ae4ae28 Configure Khoj to automatically open the browser to their web home page when Khoj is up 2023-07-01 19:46:31 -07:00
sabaimran
d2083dd395 Remove bespoke processing for GithubToJsonl file demo 2023-07-01 19:09:22 -07:00
sabaimran
a71440f62a Update the guidance in the error message if config is not set 2023-07-01 19:09:00 -07:00
sabaimran
7db97d8aa9 Fix: don't try to render the search_type.ALL 2023-07-01 19:08:19 -07:00
sabaimran
f0f6390366 Make --no-gui the default behavior of Khoj and update corresponding documentation 2023-07-01 19:07:59 -07:00
Debanjum Singh Solanky
2fbc609233 Add content write permission to jobs in github release workflow 2023-07-01 06:23:45 -07:00
Debanjum Singh Solanky
d77e05c279 Release Khoj version 0.7.0 2023-07-01 05:44:22 -07:00
Debanjum Singh Solanky
32d73500ba Update Khoj Github Plugin details in main Readme 2023-07-01 02:18:47 -07:00
Debanjum Singh Solanky
30d87a9a01 Update color of Khoj chat in Obsidinan plugin to Lantern theme 2023-07-01 02:18:47 -07:00
Debanjum Singh Solanky
51826d28d6 Ensure clicking Update in Khoj Obsidian indexes PDF files too 2023-07-01 02:18:47 -07:00
sabaimran
dac2d14380 Handle file names appropriately for md files and render commits in github results 2023-07-01 01:20:58 -07:00
sabaimran
dbe713604d Fix error in tests for markdown_to_jsonl 2023-07-01 00:49:40 -07:00
sabaimran
931aab4464 Handle case for when headers value is None 2023-07-01 00:37:30 -07:00
sabaimran
d01afb3ee4 Fix path issues for URL-based markdown files 2023-07-01 00:25:11 -07:00
sabaimran
01aa285d7b Merge pull request #260 from khoj-ai/features/add-demo-views-for-khoj 
Add demo view for Khoj
 2023-06-30 21:57:43 -07:00
sabaimran
31655447e7 Add the sign-up list to the chat page as well and update copy 2023-06-30 21:43:01 -07:00
sabaimran
cebaa51c2f Merge branch 'master' of github.com:debanjum/khoj into features/add-demo-views-for-khoj 2023-06-30 20:39:02 -07:00
sabaimran
796102c74e Add separate configuration if the given Khoj instance is meant for demo 
- In theory, this will be suitable for any Khoj instance that's meant for external-facing purposes (as in, outside of the user's network)
- Prevent re-indexing for Github data if this is a demo instance
- Fix up some issues with the CSS which made settings page small in mobile
- In the frontend views for Khoj, add a button to get on the waitlist and links to the landing page
 2023-06-30 20:38:55 -07:00
sabaimran
a443af3a71 Merge pull request #256 from khoj-ai/features/improve-telemetry 
Add additional request headers to improve telemetry
 2023-06-30 20:35:41 -07:00
sabaimran
db3026739d Resolve diffs in api.py to make /chat endpoint async with new request parameter 2023-06-30 00:25:37 -07:00
sabaimran
ef72508914 Try/catch around github file decoding, await call to search in chat API, fix img width 2023-06-30 00:23:21 -07:00
Debanjum Singh Solanky
b950889f47 Fix org-mode web renderer to handle results containing list in block 
- Break out of rendering list if at end of org block in org.js
- This would previous hang rendering results in web interface

Should try fix this upstream in org.js as well
 2023-06-29 19:01:25 -07:00
sabaimran
780c769567 Add additional request headers to improve telemetry 2023-06-29 18:51:24 -07:00
sabaimran
6c10d68262 Merge pull request #253 from khoj-ai/features/github-issues-indexing 
Support indexing Github issues as well as corresponding comments
 2023-06-29 16:02:47 -07:00
sabaimran
b2dd946c6d Rename issue to entry method for accuracy 2023-06-29 15:23:50 -07:00
Debanjum Singh Solanky
51dfa48e2b Have Khoj support Python 3.11 as Pytorch supports it now 
- Previously Khoj could only support Python upto 3.10 due to pytorch.
  But lots of folks had python 3.11 installed by default on their machines.

  This required installing python 3.10 and dealing with virtual envs.

  With Torch >= 2.0.1 now able to support python 3.11, at least one
  class of installation troubles for Khoj should drop. See
  https://github.com/pytorch/pytorch/issues/86566 for reference

- Preliminary testing indicates using the new torch 2.x may reduce
  search time by 25% (from 80ms to 60ms on Mac M1)

- Update Docs to not require mentioning python <=3.10 required
- Update Github test workflow to run khoj tests with python 3.11 too
 2023-06-29 15:13:26 -07:00
sabaimran
65bf894302 Interpret org files as a list and put them in separate divs. Update styling of search results to separate into cards 2023-06-29 15:12:48 -07:00
Debanjum Singh Solanky
d212298573 Make Configure button on web interface incrementally update by default 
We should add a way to force index everything.

But force indexing should not be the default when user is just trying
update content to index
 2023-06-29 14:52:51 -07:00
Debanjum Singh Solanky
da2de21339 Only return requested result count even if search in multiple content types 
- Set results_count to default value at start so it is an int, never None
 2023-06-29 14:49:05 -07:00
sabaimran
77672ac0ae Demarcate different results with a border box 
- Add back support for searching by type Github
- Remove custom class name in markdown js file
 2023-06-29 14:14:25 -07:00
sabaimran
6edc32f2f4 Accept current changes to include issues in rendering flow 2023-06-29 12:25:29 -07:00
Debanjum
f272d4503e Search across all Asymmetric Text Content Types in Parallel 
- Allow searching across asymmetric text content types using threads
   - Query time on my Mac averages 95ms latency (140ms at 90 percentile) across (Org, Markdown, Github, PDF and Music content types)
   - This is not too much more than search for a single content type (maybe max ~50% latency increase?). Encoding query is what takes most of the time anyway and that's just done once like before, threading adds some overhead
   - An **average** of `95 ms` latency or `140ms` at **90th percentile** is inline with keeping an incremental search (search-as-you-type) experience
- Put logic to remove filter terms from query in a `defilter` method for each filter
- Encode query once during search to encode query once across all (asymmetric) content types
- Search across all content types via the web and emacs interfaces in [d5fb419](d5fb4196de) and [5c4eb95](5c4eb950d5) respectively
- Allow Khoj Chat to pull relevant data from across content types (without the perf hit). Khoj chat is only pulling data from a single content type currently
 2023-06-29 12:21:27 -07:00
sabaimran
b41c14b258 Use *.markdown in the khoj_docker.yml 2023-06-29 11:55:18 -07:00
sabaimran
e6053951f0 In chat conftest fixtures, use *.markdown rather than *.md 2023-06-29 11:53:47 -07:00
sabaimran
ab7dabe74f Explicitly use Union type for function parameters for lint checks 2023-06-29 11:44:30 -07:00
sabaimran
601b738135 Bonus: Rename all md files to markdown for cleanliness 2023-06-29 11:27:47 -07:00
sabaimran
fecf6700d2 Limit small image rendering to just the avatar images 2023-06-29 11:27:18 -07:00
sabaimran
70e550250a Add an additional data source for issues from Github repositories + quality of life updates 
- Use a request session to reduce the overhead of setting up a new connection with the Github URL each request
- Use the streaming feature for the REST api to reduce some of the memory footprint
 2023-06-29 10:59:54 -07:00
Debanjum Singh Solanky
5f2717cc4b Use logger.warning since logger.warn is deprecated 2023-06-28 22:15:27 -07:00
Debanjum Singh Solanky
5f7eaa7ded Add trio, move freezegun, factory-boy to project test dependencies 2023-06-28 22:07:02 -07:00
Debanjum Singh Solanky
56ce97ef9e Use async/await in tests for query method of text and image search 
The text, image search query method has become async. So async/await
is required to get results correctly in tests etc
 2023-06-28 22:07:02 -07:00
Debanjum Singh Solanky
f516d127c8 Update client tests to expect "all" as a valid new content type 2023-06-28 22:07:02 -07:00
Debanjum Singh Solanky
b1767f93d6 Get any configured asymmetric search model to encode query for search 
- Set image_search.query to async to use it with multi-threading
  This is same as text_search.query being set to an async method
- Exit search early if no search_model is defined in state.model
 2023-06-28 22:07:02 -07:00
Debanjum Singh Solanky
8eae7c898c Put each result under org heading when query for "all" content type in khoj.el 
- Add "all" as default content type when no content type retrieved
  from server
 2023-06-28 22:07:02 -07:00
Debanjum Singh Solanky
630bf995f1 Style each result based on its content type in same view on Khoj web 
- So when searching across content types (with content-type = "all")
  org-mode results get rendered differently than markdown, PDF etc. results

- Set div class for each result separately instead of a single uber div
  for styling. This allows styling div of each result based on the
  content-type of that result

- No need to create placeholder "all" content type on web interface as
  server is passing an all content type by itself
 2023-06-28 22:07:01 -07:00
Debanjum Singh Solanky
1773a78339 Fix createRequestUrl method signature to fetch results from khoj web 2023-06-28 12:10:45 -07:00
Debanjum Singh Solanky
212b1a96c8 Create "all" search type for search across all content types on khoj server 
Allows moving logic to handle search across all content types to
server from clients
 2023-06-28 11:34:26 -07:00
Debanjum Singh Solanky
0636ceaf14 Merge branch 'master' of github.com:khoj-ai/khoj into parallelize-search-across-all-asymmetric-text-content-types 
Conflicts:
- src/khoj/routers/api.py: Use theirs
 2023-06-27 16:10:32 -07:00
Debanjum Singh Solanky
510bb7e684 Use typing union in text_search for python 3.8 compatible type hinting 2023-06-27 15:59:50 -07:00
Debanjum Singh Solanky
1b11d5723d Extract search request URL builder into js function in web interface 2023-06-27 15:50:41 -07:00
Debanjum Singh Solanky
09f739b8cc Null check config, log warning instead of error when configuring search 2023-06-27 15:48:48 -07:00
sabaimran
c0d35bafdd Merge pull request #250 from khoj-ai/features/github-multi-repo-and-more 
Support multiple Github repositories and support indexing of multiple file types
 2023-06-27 15:14:49 -07:00
sabaimran
9d62d66a77 Simplify construction of repo shorthand in GithubToJsonl 2023-06-27 15:05:03 -07:00
sabaimran
2697c7a186 Update org tests to use new method, update Github configuration in tests 2023-06-27 15:04:48 -07:00
sabaimran
227169ebde Support configuration of multiple Github repositories in the settings interface 
- Add cards to configure each of the Github repositories
- Fix a bug in the API which caused all other settings to be wiped when updating one of the content types
- Provide an error message to the user if they have a misconfiguration in their chat settings
 2023-06-27 14:10:09 -07:00
sabaimran
37a1f15c38 Add backend support for indexing multiple repositories 
- Add support for indexing org files as well as markdown files from the Github repository and update corresponding search view
- Support indexing a list of repositories
 2023-06-27 12:06:15 -07:00
Debanjum Singh Solanky
5da6a5e669 Build docker image using latest khoj from git master 
- Previous state
  Ideally docker image should use latest app code available locally.
  But this is better than the previous state where the latest Docker
  image was being built using older khoj package published to pypi

  This would happen because the workflow to publish the khoj-assistant
  pypi package runs in parallel to the dockerize workflow so the latest
  khoj pypi package isn't published before the latest docker image is
  built on master

- Updated state
  Now at least the docker image published via the dockerize github
  workflow will be built using the latest khoj code on github
 2023-06-26 20:16:07 -07:00
sabaimran
ddd550e6f4 Add call to use X-CSRFToken in relevant POST methods 2023-06-26 12:38:00 -07:00
sabaimran
35e24d7851 Fix null checking in state for content config API and telemetry API 2023-06-26 11:37:34 -07:00
sabaimran
5e39421f56 Merge branch 'master' of github.com:debanjum/khoj 2023-06-25 11:41:47 -07:00
sabaimran
4410a3bb4b Limit max width of the pre tag to 100% of the screen width 2023-06-25 11:41:15 -07:00
sabaimran
ffe66b848a Use a single column tempalte for config plugins when in mobile 2023-06-25 11:27:41 -07:00
Debanjum Singh Solanky
b1890aa050 Null check intermediary objects when config not fully initialized 2023-06-24 15:34:18 -07:00
Debanjum Singh Solanky
946af0889d Improve showing status message on saving config via web interface 
- Show success/failure status message much closer to the save button
  Previously status message was shown on top of the page, which wasn't
  always in view and wasn't easily seen
- Improve the status message to more clearly show next steps on success
 2023-06-24 00:49:57 -07:00
Debanjum Singh Solanky
40d1abfe50 Update the new /config APIs to configure Khoj for first time users 
- Setup state.config and sub-components from unset state
- Setup search types with default settings
 2023-06-24 00:45:30 -07:00
Debanjum Singh Solanky
05a3c81adb Add beautiful as dependency to pass pytests 2023-06-23 15:10:09 -07:00
Debanjum Singh Solanky
edabede93a Fix post configuration state update on error or success on config html 2023-06-23 14:52:25 -07:00
Debanjum
98642e01b5 Update Web Interface with Lantern Theme 
- Style all pages with consistent lantern theme styling
  - Add navigation pane to all web interface pages
  - a200af68b38d0625c42e2098d171c6ddab121bd2 Keep pico.css locally for offline usage
  - cd8d069e6673b4db4c14f736c3d8af80bf94614d Highlight currently active tab in web interface
- Update config pages to use Lantern theme
 2023-06-23 14:39:25 -07:00
Debanjum Singh Solanky
4744d69221 Resolve button name, anchor tag feedback. Add status message to settings page 
- Use "Configure" name for settings config action
- Use more standard anchor tag instead of button
- Add configure status message
 2023-06-23 09:48:38 -07:00
Debanjum Singh Solanky
26abafa658 Highlight currently active tab in web interface for orientation 2023-06-22 00:33:28 -07:00
Debanjum Singh Solanky
2728c714d7 Put pico.css in local assets. Move common css styling into khoj.css 2023-06-22 00:33:11 -07:00
Debanjum Singh Solanky
20a37697de Add Khoj header with navigation pane to Search and Chat Interfaces 2023-06-22 00:33:11 -07:00
Debanjum Singh Solanky
c467a0cbb0 Update UI of config sub pages to use khoj lantern theme styling 2023-06-22 00:33:11 -07:00
Debanjum Singh Solanky
0ce2ec590a Update main config page on khoj server to match khoj lantern theme 2023-06-21 20:25:25 -07:00
Debanjum Singh Solanky
d30a9ddd33 Use Khoj Logo on Search, Chat pages of Web Interface 2023-06-21 12:34:53 -07:00
Debanjum Singh Solanky
6d4aad57e1 Use new Khoj Lantern Logo in Web, Emacs, Obsidian UIs and Docs 2023-06-21 01:57:22 -07:00
Debanjum Singh Solanky
69d4fa6525 Rename project links across repo from debanjum/khoj to khoj-ai/khoj 2023-06-21 00:13:21 -07:00
Debanjum Singh Solanky
5c4eb950d5 Search across all content types via khoj.el on Emacs 
If no content-type selected in transient menu option, khoj.el queries
khoj server without content-type parameter (t) set.

This results in search across all enabled asymmetric search text
content types
 2023-06-20 23:39:56 -07:00
Debanjum Singh Solanky
2cd3e799d3 Improve null and type checks 2023-06-20 23:30:59 -07:00
Debanjum Singh Solanky
d5fb4196de Update web interface to allow querying all content types at once 2023-06-20 22:21:50 -07:00
Debanjum Singh Solanky
5c7c8d1f46 Use async/await to fix parallelization of search across content types 2023-06-20 22:21:50 -07:00
Debanjum Singh Solanky
1192e49307 Pass default value matching argument types expected by text_search methods 2023-06-20 22:21:50 -07:00
Debanjum Singh Solanky
0144e610d6 Only search across content types that work with asymmetric search 2023-06-20 22:21:46 -07:00
Debanjum Singh Solanky
f6a7aa6c96 Style Khoj chat on web interface with new lantern theme 
- Color khoj chat message with new yellow theme color
- Update Khoj chat emoji to lantern
- Add page type to title of pages on web interface
 2023-06-20 01:39:33 -07:00
Debanjum Singh Solanky
6d94d6e75a Encode the asymmetric, symmetric search queries in parallel for speed 
Use timer to measure time to encode queries and total search time
 2023-06-20 01:18:17 -07:00
Debanjum Singh Solanky
d292dc03b3 Use new Khoj Logotype in Web interface 2023-06-20 01:13:06 -07:00
Debanjum Singh Solanky
db07362ca3 Encode user query as same across search types to speed up query time 
- Add new filter abstract method to remove filter terms from query
- Use the filter method to remove filter terms, encode this defiltered
  query and pass it to the query methods of each search types

TODO: Encoding query is still taking 100-200 ms unlike before. Need to
investigate why
 2023-06-19 23:29:54 -07:00
Debanjum Singh Solanky
285d17af2a Search in parallel across all enabled content types requested via API 
- Update API to return content from all enabled content types when type
  is not set to specific type in HTTP request param
- To do this efficiently run the search queries in parallel threads
 2023-06-19 23:29:06 -07:00
Debanjum Singh Solanky
79d325fbb6 Fix triggering @general queries in Khoj Chat 2023-06-19 23:05:33 -07:00
Debanjum Singh Solanky
e97a20d70c Set conversation type if query param set, else return chat history 
Only initialize variables if query is not empty, to avoid unnecessary
compute, variable null checks etc.

Fixes #230
 2023-06-19 19:59:16 -07:00
sabaimran
6224dce49d Merge pull request #228 from debanjum/features/pretty-config-page 
Update the config page to be more usable
 2023-06-19 18:11:35 -07:00
sabaimran
4722a2c16d Add Github configuration page and success notifications 2023-06-18 10:06:45 -07:00
sabaimran
668135c763 Merge branch 'master' of github.com:debanjum/khoj into features/pretty-config-page 2023-06-18 08:35:09 -07:00
sabaimran
81183a1fe1 Address misc PR comments and update logo in all clients 
- Rename the new logo to reflect accuracy on size (e.g., 128x128)
- Update the icns file for Mac
- Update nomenclature in settings pages
 2023-06-18 08:34:58 -07:00
Debanjum Singh Solanky
a44cde2865 Show hint to re-index vault if wonky results in Obsidian search modal 
Remove spurious indentation in Obsidian styles.css

Resolves #207
 2023-06-18 04:53:51 -07:00
Debanjum Singh Solanky
595cc5b0f5 Use printer icon for PDF logs. Only split lines if file at web link in web interface 2023-06-18 02:26:03 -07:00
Debanjum
e06be395f9 Use Github REST API and Index Commit Messages off Github Repository 
- Migrate to Github REST API instead of Llama Hub to index Markdown Docs in Github Repository
- Index Commit Messages from Github Repository as well
 2023-06-18 14:51:32 +05:30
Debanjum Singh Solanky
e31a540a5e Get all md files recursively in repository by passing recursive param 
Previously the `get_markdown_files' method was only getting files at
root of the repository

Fix, improve logger messages in github to jsonl processor
 2023-06-18 01:47:15 -07:00
Debanjum Singh Solanky
6fdac24416 Set page size to 100 to reduce requests required to Github API to 1/3 
- Default is 30. So number of paginated requests required to get all
  items (commits, files) will reduce by 67%

- No need to increase page size for the get tree Github API request from
  `get_markdown_files'

  Get tree Github API doesn't support pagination and return 100K items
  in response. This should be way more than enough for our current
  use-cases
 2023-06-18 01:44:36 -07:00
Debanjum Singh Solanky
87975e589a Fix passing auth token to Github API to increase rate limits by x85 
- Previously wasn't prefixing "token" to PAT token in Auth header
  This resulted in the request being considered unauthenticated

- Unauthenticated requests to Github API are limited to 60 requests/hour
  Authenticated requests to Github API are allowed 5000 requests/hour
 2023-06-18 01:19:26 -07:00
Debanjum Singh Solanky
9c70af960c Extract logic to get file content from Github into a separate method 2023-06-18 01:19:13 -07:00
Debanjum Singh Solanky
10d4c38ce9 Extract Wait for rate limit reset logic into a function for reuse 2023-06-18 01:06:46 -07:00
sabaimran
aad7f825e0 Remove music configuration 2023-06-17 21:23:56 -07:00
sabaimran
5f97afbfac Ignore type checks from mypy in subindexed fields 2023-06-17 16:53:36 -07:00
sabaimran
c2d46de8bc Add endpoint for regenerating directly from the config page and add music content-type 2023-06-17 15:47:33 -07:00
sabaimran
ded3100caf Update the configuration page to make config management easier 
- Add a central configuration management page to make management of config details easier
- Add relevant api endpoints both for client and server to update/request data as necessary
- Attempt to update the favicon
 2023-06-17 15:21:28 -07:00
Debanjum Singh Solanky
3f24e53b6e Render URL as link in web interface if file param of result is a web link 2023-06-17 04:26:40 -07:00
Debanjum Singh Solanky
63ec84ad78 Store Github URL of Markdown files on Github in file jsonl param 2023-06-17 04:23:01 -07:00
Debanjum Singh Solanky
0c1c7583b5 Handle pagination, API rate limits. Get all commits from Github repo 2023-06-17 04:21:39 -07:00
Debanjum Singh Solanky
31d17d0b22 Index commits message from repository with the github plugin 2023-06-17 02:59:54 -07:00
Debanjum Singh Solanky
c29c141a7e Use Github Rest API to index Markdown files in Github Repository 
The Llama_Hub Github plugin is fairly limited.

The Github Rest API is well supported and can easily be extended to
index commit messages, issues, discussions, PRs etc.
 2023-06-17 02:16:13 -07:00
Debanjum
9f00a366ab Add a Github plugin to index content from a Github repository 
- Use the Github plugin on LlamaHub to read in markdown files from specified Github repository for indexing
- Update the desktop GUI application to take in the required parameters to read from Github
- Requires a classic PAT token for Github access
 2023-06-17 12:28:47 +05:30
Saba
ac96f43b1b Remove try-catch specific to Github plugin; consolidate GUI logic 2023-06-16 23:46:25 -07:00
Saba
07ade2262a Set default value of pat_token in conftest.py to be empty string 2023-06-13 17:03:03 -07:00
Saba
751edfefe5 Add separate unit test for github. Will only run of a PAT token is set 2023-06-13 16:55:58 -07:00
Saba
3a61919344 Fix failing unit tests by hard-coding model presence of expected search types 2023-06-13 16:32:47 -07:00
Saba
019d3732de Rename orgmode_search to org_search 2023-06-13 16:06:54 -07:00
Saba
08d79f5ba4 Unify types used in Github and other text-based configs. Fix typing issues 2023-06-13 15:52:36 -07:00
Saba
a6cd96a6a9 Add a Github plugin which can be used to read from a Github repository 2023-06-13 14:40:06 -07:00
Debanjum
c68cde4803 Log clients calling API endpoints on Khoj server 
- Make API endpoints on Khoj server accept `client` as request parameter
  - Khoj API endpoints: /chat, /search, /update
- Make Khoj clients set `client` request param when calling the API endpoints on the Khoj server
  - Khoj clients: Emacs, Obsidian and Web
- Also log khoj server_version running to telemetry server
 2023-06-09 18:36:49 +05:30
sabaimran
59fa48036f Merge pull request #224 from debanjum/fix/message-exceeds-prompt-size 
Pass truncated message as string in ChatMessage when exceeding max prompt size
 2023-06-08 17:32:53 -07:00
Debanjum Singh Solanky
139a3ba060 Update server to log new server version field to telemetry db 2023-06-08 14:14:21 +05:30
Saba
c5666e0404 Move factory dependencies to optional settings 2023-06-06 23:26:24 -07:00
Saba
5d5ebcbf7c Rename truncate messages method and update unit tests to simplify assertion logic 2023-06-06 23:25:43 -07:00
Saba
7119ed0849 Run pre-commit script 2023-06-05 19:29:23 -07:00
Saba
948ba6ddca Remove unused logger 2023-06-05 19:01:03 -07:00
Saba
6212d7c2e8 Remove debug line 2023-06-05 19:00:25 -07:00
Saba
f65ff9815d Move message truncation logic into a separate function. Add unit tests with factory boy. 2023-06-05 18:58:29 -07:00
Debanjum Singh Solanky
eb6175e9b0 Update description field in webmanifest of Khoj, Khoj Chat PWA 2023-06-06 01:53:42 +05:30
Debanjum Singh Solanky
bb2363f324 Set client request param when calling khoj server APIs from Web 2023-06-06 00:05:00 +05:30
Debanjum Singh Solanky
caab55fbdd Set client request param when calling khoj server APIs from Obsidian 2023-06-06 00:04:46 +05:30
Debanjum Singh Solanky
de2494154f Set client request param when calling khoj server APIs from Emacs 2023-06-06 00:02:10 +05:30
Debanjum Singh Solanky
168c11cea7 Make server API endpoints accept client as query param 
- The chat, search and update API will accept client as request param.
- This will allow logging the client from which these APIs was called.
 2023-06-05 23:57:08 +05:30
Debanjum Singh Solanky
8617cf1389 Push telemetry to Posthog to grok Khoj usage 2023-06-05 22:47:49 +05:30
Debanjum Singh Solanky
d13db2e666 Make old telemetry server forward requests to new server 2023-06-05 13:06:45 +05:30
Saba
5f4223efb4 Increase timeout to OpenAI call 2023-06-04 20:49:47 -07:00
Saba
0e63a90377 Fix the mechanism to retrieve the message content 2023-06-04 20:25:37 -07:00
Saba
f0efe0177e Pass truncated message as string in ChatMessage when exceeding max prompt size 2023-06-04 19:33:46 -07:00
Debanjum
f6ceb22373 Use api_key keyword argument to set the openai_api_key parameter for GPT 2023-06-04 15:05:34 +05:30
Saba
068ee0ac5e Swap elif with else, as usage of this method does not use openai_api_key 2023-06-04 02:25:08 -07:00
Saba
6508379d7b Use api_key keyword argument to set the openai_api_key parameter for GPT 2023-06-04 00:57:00 -07:00
Debanjum Singh Solanky
7af8a56434 Remove filename from reference before rendering references in khoj.el 
Fixes bug where actual reference heading in next line jumping out of
references footnote section
 2023-06-02 10:42:44 +05:30
Debanjum Singh Solanky
ec280067ef Do not retrieve relevant notes when having a general chat with Khoj 
- This improves latency of @general chat by avoiding unnecessary
  compute
- It also avoids passing references in API response when they haven't
  been used to generate the chat response. So interfaces don't have to
  add logic to not render them unnecessarily
 2023-06-02 10:42:44 +05:30
Debanjum Singh Solanky
90439a8db1 Update Khoj subtitle to AI personal assistant for your digital brain 2023-06-02 10:42:44 +05:30
Debanjum
e022910f31 Search PDF files with Khoj. Integrate with LangChain 
- **Introduce Khoj to LangChain**: 
    Call GPT with LangChain for Khoj Chat
- **Search (and Chat about) PDF files with Khoj**
  - Create PDF to JSONL Processor: Convert PDF content into standardized JSONL format
  - Expose PDF search type via Khoj server API
  - Enable querying PDF files via Obsidian, Emacs and Web interfaces
 2023-06-02 10:20:26 +05:30
Debanjum Singh Solanky
e9ed7a19fd Update search prompt to extract PDF search type. Fix extract_question prompt 2023-06-02 10:06:03 +05:30
Debanjum Singh Solanky
89fbfce20a Mention PDF are also supported in Khoj Readme 2023-06-01 21:42:48 +05:30
Debanjum Singh Solanky
bbe3bf9733 Render PDF search results in Khoj Obsidian interface 
- Make plugin update khoj server config to index PDF files in vault too
- Make Obsidian plugin update index for PDF files in vault too
- Show PDF results in Khoj Search modal as well
  - Ensure combined results are sorted by score across both types
- Jump to PDF file when select it PDF search result from modal
 2023-06-01 21:42:48 +05:30
Debanjum Singh Solanky
e3892945d4 Render PDF search results in Khoj.el Emacs interface 2023-06-01 21:42:48 +05:30
Debanjum Singh Solanky
85144006a1 Render PDF search results in khoj web interface 2023-06-01 21:42:48 +05:30
Debanjum Singh Solanky
acd14a5e41 Wire up PDF to jsonl processor to Khoj server layer (API, config) 
- Specify PDF content to index via khoj.yml
- Index PDF content on app start, reconfigure
- Expose PDF as a search type via API
 2023-06-01 21:42:48 +05:30
Debanjum Singh Solanky
d63194c3a9 Create tests for PDF to JSONL processor 2023-06-01 21:42:48 +05:30
Debanjum Singh Solanky
286b500f66 Create PDF to JSONL processor using PyPDF and LangChain 
Switch `pydantic' to >= 1.9.1 else `langchain.document_loaders' starts
throwing typing error for python 3.8, 3.9
 2023-06-01 21:41:49 +05:30
Debanjum Singh Solanky
1b3effd8e6 Fork Markdown to JSONL processor as start template for PDF to Jsonl Processor 2023-06-01 09:13:31 +05:30
Debanjum Singh Solanky
1cd9ecd449 Truncate last message if still over max supported prompt size by model 2023-06-01 08:50:59 +05:30
Debanjum Singh Solanky
ed4d0f9076 Simplify argument names used in khoj openai completion functions 
- Match argument names passed to khoj openai completion funcs with
  arguments passed to langchain calls to OpenAI
- This simplifies the logic in the khoj openai completion funcs
 2023-06-01 08:50:59 +05:30
Debanjum Singh Solanky
703a7c89c0 Reduce retry count and request timeout for faster response or failure 
- Fix bug where both LangChain and Khoj retry requests 6 times each.
  So a total of 12 requests at >1minute intervals for each chat
  response in case of OpenAI API being down

- Retrying too many times when the API is failing doesn't help
- The earlier 60 second request timeout was spacing out the interval
  between retries way too much. This slowed down chat response times
  quite a bit when API was being flaky

- With these updates you'll know if call to chat API failed in under a
  minute
 2023-06-01 08:50:59 +05:30
Debanjum Singh Solanky
18081b3bc6 Use LangChain to call GPT over API 2023-06-01 08:50:59 +05:30
Debanjum Singh Solanky
277d2f5c96 Do not add "Notes:" suffix to chat messages when no notes retrieved 
This was causing spurious "Notes:" suffix being added to Khoj Chat in
response
 2023-06-01 08:50:59 +05:30
Debanjum Singh Solanky
334be4e600 Use LangChain to call OpenAI for Khoj Chat 
- Use ChatModel and ChatOpenAI to call OpenAI chat model instead of
  using OpenAI package directly
- This is being done as part of migration to rely on LangChain for
  creating agents and managing their state
 2023-06-01 08:50:59 +05:30
Debanjum Singh Solanky
efcf7d1508 Extract prompts as LangChain Prompt Templates into a separate module 
Improves code modularity, cleanliness. Reduces bloat in GPT.py module
 2023-06-01 08:50:58 +05:30
Debanjum Singh Solanky
b484953bb3 Import app state correctly to generate embeddings with OpenAI model 
Resolves #216
 2023-05-28 10:21:54 +05:30
Debanjum Singh Solanky
9cfaaf0941 Update docs to configure khoj.yml for using OpenAI model for embeddings 2023-05-28 10:21:54 +05:30
Debanjum Singh Solanky
a0d0dbaca7 Fix link to Khoj Obsidian Demo video in Readmes 2023-05-23 04:23:08 +05:30
Debanjum Singh Solanky
ebb5d7b8e5 Release Khoj version 0.6.2 2023-05-17 20:04:20 +05:30
Debanjum Singh Solanky
d02415edcc Write generated server id to env file when env file does not contain it 2023-05-17 19:38:44 +05:30
Debanjum Singh Solanky
dc0626856e Put the telemetry db in a separate directory by default 2023-05-17 18:58:47 +05:30
Debanjum
dc495babb3 Add Telemetry to Understand Khoj Usage 
### Objective: 
Use telemetry to better understand Khoj usage.
This will motivate and prioritize work for Khoj.

Specific questions:
- Number of active deployments of khoj server
- How regularly is khoj used (hourly, daily, weekly etc)?
- How much is which feature used (chat, search)?
- Which UI interface is used most (obsidian, emacs, web ui)?

### Details
- Expose setting to disable telemetry logging in khoj.yml
- Create basic telemetry server to log data to a DB
- Log calls to Khoj API /search, /chat, /update endpoints
- Batch upload telemetry data to server at ~hourly interval
 2023-05-17 19:09:50 +08:00
Debanjum Singh Solanky
55d72231b3 Generate docker image for telemetry server using Github workflow 2023-05-17 16:08:21 +05:30
Debanjum Singh Solanky
e9f04dc644 Add dockerfile to containerize telemetry server 2023-05-17 16:08:21 +05:30
Debanjum Singh Solanky
07b19964d4 Schedule jobs at (co-)prime intervals to reduce overlap in job runs 2023-05-17 16:08:21 +05:30
Debanjum Singh Solanky
d42f0f5055 Add basic telemetry server for khoj 2023-05-17 16:08:21 +05:30
Debanjum Singh Solanky
134cce9d32 Batch upload telemetry data at regular interval instead of while querying 2023-05-17 16:08:21 +05:30
Debanjum Singh Solanky
3ede919c66 Log usage of /search, /chat, /update API endpoints to telemetry server 2023-05-17 16:08:21 +05:30
Debanjum Singh Solanky
f2e89f6f46 Add khoj app helper methods to log app usage to a telemetry server 2023-05-17 16:08:21 +05:30
Debanjum Singh Solanky
9ca61d62ff Enable/disable logging telemetry by setting bool in khoj.yml config 
We log usage telemetry by default, unless setting explicitly set in
khoj.yml
 2023-05-15 23:26:38 +08:00
Debanjum Singh Solanky
131b8407b5 Allow Khoj Chat to respond to general queries not in reference notes 
- Khoj chat will now respond to general queries if:
  1. no relevant reference notes available or
  2. when explicitly induced by prefixing the chat message with "@general"

- Previously Khoj Chat would a lot of times refuse to respond to
  general queries not answerable from reference notes or chat history

- Make chat quality tests more robust
  - Add more equivalent chat response options refusing to answer
  - Force haiku writing to not give any preable, just the haiku
 2023-05-12 18:42:40 +08:00
Debanjum Singh Solanky
cc75f986b2 Test text search index only updates on changes to text content 2023-05-12 17:37:34 +08:00
Debanjum Singh Solanky
f9ccce430e Allow configuring OpenAI chat model for Khoj chat 
- Simplifies switching between different OpenAI chat models. E.g GPT4
- It was previously hard-coded to use gpt-3.5-turbo. Now it just
  defaults to using gpt-3.5-turbo, unless chat-model field under
  conversation processor updated in khoj.yml
 2023-05-03 23:01:13 +08:00
Debanjum
f0253e2cbb Include Filename, Entry Heading in All Compiled Entries to Improve Search Context 
Merge pull request #214 from debanjum/add-filename-heading-to-compiled-entry-for-context

- Set filename as top heading in compiled org, markdown entries
  - Note: *Khoj was already indexing filenames in compiled markdown entries but they weren't set as top level headings but rather appended as bare text*. The updated structure should provide more schematic context of relevance
- Set entry heading as heading for compiled org, md entries, even if split by max tokens
- Snip prepended heading to avoid crossing model max_token limits
- Entries with no md headings should not get heading prefix prepended
 2023-05-03 22:59:30 +08:00
Debanjum Singh Solanky
6b535cc345 Snip prepended heading to avoid crossing model max_token limits 
Otherwise if heading > max_tokens than the search models will just see
a heading (with repeated filename) for each compiled entry and not
actual content.

100 characters should be sufficient to include filename (not path) and
entry heading. If longer rather truncate to pass entry unique text to
model for search context
 2023-05-03 22:53:13 +08:00
Debanjum Singh Solanky
02aeee60aa Set filename as top heading of org entries for better search context 
Previously filename was only being appended to markdown entries.

Test filename getting prepended to compiled entry as heading
 2023-05-03 22:53:13 +08:00
Debanjum Singh Solanky
94825a70b9 Set heading of md entries to improve search context for long entries 
Otherwise if a markdown entry is longer than max_tokens, the split
entries (apart from first one) do not get their heading context set
 2023-05-03 22:53:13 +08:00
Debanjum Singh Solanky
5de04621b5 Set filename as top heading of md entries for better search context 
Previously filename was appended to the end of the compiled entry.
This didn't provide appropriate structured context

Test filename getting prepended as heading to compiled entry
 2023-05-03 22:50:31 +08:00
Debanjum Singh Solanky
0e3fb59e09 Entries with no md headings should not get heading prefix prepended 
Files with no headings would previously get their entry be prefixed
with a markdown heading prefix (#)
 2023-05-03 22:50:31 +08:00
Debanjum Singh Solanky
45a991d75c Prepend entry heading to all compiled org snippets to improve search context 
All compiled snippets split by max tokens (apart from first) do not
get the heading as context.

This limits search context required to retrieve these continuation
entries
 2023-05-03 22:50:31 +08:00
Debanjum Singh Solanky
3386cc92b5 Fix khoj server config update in khoj.el by unquoting list to cl-push to 
- cl-push expects a generatlized variable. Else throws (setf quote)
  undefined warning
- This results in the config call failing on calling khoj entrypoint
 2023-05-03 15:10:56 +08:00
Debanjum Singh Solanky
948a4274e4 Fix documentation strings and simplify not null checks 2023-05-02 21:47:50 +08:00
Debanjum Singh Solanky
731ef5688f Use cl-pushnew to fix byte-compile errors with using add-to-list 2023-05-02 21:47:38 +08:00
Debanjum Singh Solanky
f046523b33 Improve khoj.el messages to convey state of khoj server 
- Remove waiting for server message as it hides the messages from the
  server
- Fix the nil message that were being rendered, by checking before
  showing messages from server
- Consistently prefix messages from khoj with khoj.el
 2023-04-28 11:15:13 +08:00
Debanjum Singh Solanky
76df393eb5 Only call khoj server configure API from khoj.el when config updated 
Previously khoj.el was calling the server configure API even when
config was same as before.
This had broken the khoj search as you type experience from emacs

Also show more details to user about what in khoj is being configured
 2023-04-27 20:45:16 +08:00
Debanjum Singh Solanky
ceae06ae9d Fix khoj.el compilation warnings around unused variables 2023-04-27 20:45:16 +08:00
Debanjum Singh Solanky
8269adf849 Refactor khoj-setup in khoj.el for readability. No functional change 2023-04-27 20:45:00 +08:00
Debanjum Singh Solanky
865d12b6f2 Fix escaping quote in chat references to prevent it breaking out of html 2023-04-27 20:45:00 +08:00
Debanjum Singh Solanky
26cb878327 Add Yarn lockfile for Khoj Obsidian 2023-04-18 00:57:11 +07:00
Debanjum Singh Solanky
e3180d63e6 Sync Khoj Obsidian Tagline with Khoj tagline 2023-04-18 00:56:50 +07:00
Debanjum Singh Solanky
62e6e09521 Release Khoj version 0.6.1 2023-04-17 23:31:35 +07:00
Debanjum Singh Solanky
b079fb31bc Replace Windows path separators in indexName configured via Khoj Obsidian 
Resolves #185, #199

- Issue
  IndexName created from Obsidian Absolute Vault path wasn't replacing
  windows path, drive separators with underscore. It was only
  replacing unix path separators

- Fix
  Also replace windows drive and path separators with _ while creating
  IndexName in Khoj Obsidian plugin
 2023-04-17 16:55:33 +07:00
Debanjum Singh Solanky
d90df966a9 Make khoj logger use utf-8 encoding when writing to khoj log file 
Resolve logger error issue mentioned in #199
 2023-04-17 16:55:07 +07:00
Debanjum Singh Solanky
dc3f399f91 Fix to get score associated with SearchResponse in result as string 2023-04-16 20:22:51 +07:00
Debanjum Singh Solanky
d5000c63e1 Update Readmes to use python -m pip install khoj-assistant 
Makes it easier to tell pip associated with which python is being
used. Easier to debug when users have different versions of python
installed (e.g 3.10 and 3.11)
 2023-04-16 20:17:20 +07:00
Debanjum Singh Solanky
453c84ab79 Add Screenshots of Khoj Chat Interface on Emacs, Obsidian to Readmes 2023-04-07 23:19:47 +07:00
Debanjum Singh Solanky
35aa06067f Release Khoj version 0.6.0 
Upload styles.css via release workflow
 2023-03-31 18:13:16 +07:00
Debanjum
8f4e5d3d83 Improve Styling of Khoj Search Modal on Obsidian and Indexing of Markdown 
Merge pull request #198 from debanjum/improve-khoj-search-for-markdown-obsidian

### Overview
- Copied Khoj Search Modal styling from Jim Prince's PR #135 with minor improvements
- Implements improvements to the Khoj Search in Markdown/Obsidian suggested by folks. Specifically:
  - #133
  - #134
  - #142

### Changes
- 5673bd5 Keep original formatting in compiled text entry strings
- a2ab68a Include filename of markdown entries for search indexing
- 6712996 Create Note with Query as title from within Khoj Search Modal
- d3257cb Style the search result. Use Obsidian theme colors and font-size
- 4009148 For each result: snip it by lines, show filename, remove frontmatter
 2023-03-30 14:15:23 +07:00
Debanjum Singh Solanky
5673bd5b96 Keep original formatting in compiled text entry strings 
- Explicity split entry string by space during split by max_tokens
- Prevent formatting of compiled entry from being lost
- The formatting itself contains useful information
  No point in dropping the formatting unnecessarily,
  even if (say) the currrent search models don't account for it (yet)
 2023-03-30 14:02:46 +07:00
Debanjum Singh Solanky
a2ab68a7a2 Include filename of markdown entries for search indexing 
Append originating filename to compiled string of each entry for
better search quality by providing more context to model

Update markdown_to_jsonl tests to ensure filename being added

Resolves #142
 2023-03-30 13:51:36 +07:00
Debanjum Singh Solanky
67129964a7 Create Note with Query as title from within Khoj Search Modal 
This follows expected behavior for obsidain search modals
E.g Ominsearch and default Obsidian search.

The note creation code is borrowed from Omnisearch.

Resolves #133
 2023-03-30 13:51:36 +07:00
Debanjum Singh Solanky
d3257cb24e Style the search result. Use Obsidian theme colors and font-size 
Based on PR #135
 2023-03-30 12:35:29 +07:00
Debanjum Singh Solanky
40091489c0 For each result: snip it by lines, show filename, remove frontmatter 
Based on PR #135
Resolves #134
 2023-03-30 12:34:55 +07:00
Debanjum Singh Solanky
240db7b4f0 Add screenshot of Khoj chat on Obsidian to Readme. Fix links 2023-03-30 02:49:05 +07:00
Debanjum Singh Solanky
234be96e53 Fix processor key used to configure chat model in khoj obsidian 2023-03-30 01:47:09 +07:00
Debanjum
53d421f9c6 Create Chat Modal for Obsidian Plugin 
Merge pull request #196 from debanjum/create-chat-modal-for-obsidian

- Set your OpenAI API key in the Khoj Obsidian Settings
- Use Modal in Obsidian for Chat
- Style Chat Modal combining the Khoj Web interface and Obsidian theme style
 2023-03-30 01:37:07 +07:00
Debanjum Singh Solanky
c8c0cfd10e Add Chat features, setup and usage to Khoj Obsidian plugin Readme 2023-03-30 00:32:24 +07:00
Debanjum Singh Solanky
7ecae224e7 Configure OpenAI API Key from the Khoj plugin setting in Obsidian 2023-03-29 23:54:08 +07:00
Debanjum Singh Solanky
3d616c8d65 Use Obsidian font sizes. Improve input field, reference indexing 
- Give space in the input field. Too narrow previously
- References should be indexed from 1 instead of 0
- Use Obsidian font size variables to scale fonts in chat appropriately
 2023-03-29 22:13:55 +07:00
Debanjum Singh Solanky
23bd737f6b Use chat input element to send message on Enter. No send button required 2023-03-29 22:13:30 +07:00
Debanjum Singh Solanky
81e98c3079 Scroll to bottom of modal on open and message send 2023-03-29 18:12:12 +07:00
Debanjum Singh Solanky
59ff1ae27f Use obsidian theme colors for bg, text. Restrict css namespace via prefix 2023-03-29 18:12:12 +07:00
Debanjum Singh Solanky
001ac7b5eb Style Obsidian Chat Modal like Khoj Chat Web Interface 
- Add message sender, date metadata as message footer
- Use css directly from Khoj Chat Web Interface.
  - Modify it to work under a Obsidian modal
  - So replace html, body styling from web interface to instead
    styling new "khoj-chat" class attached to contentEl of modal
 2023-03-29 18:12:12 +07:00
Debanjum Singh Solanky
112f388ada Render references next to chat responses by khoj in chat modal 2023-03-28 18:11:03 +07:00
Debanjum Singh Solanky
1d3d949962 Render conversation logs on page load 2023-03-28 14:56:29 +07:00
Debanjum Singh Solanky
cd46a17e5f Add Khoj Chat Modal, Command in Khoj Obsidian to Chat using API 2023-03-28 14:56:29 +07:00
Debanjum Singh Solanky
c0972e09e6 Rename KhojModal to KhojSearchModal, a more specific name for it 
In preparation to introduce Khoj chat in Obsidian
 2023-03-28 14:56:29 +07:00
Debanjum Singh Solanky
64fff1d372 Release Khoj version 0.5.0 2023-03-28 03:35:59 +07:00
Debanjum Singh Solanky
7478d08803 Update main readme to mention chat features 2023-03-27 22:02:53 +07:00
Debanjum Singh Solanky
fc218508f9 Update khoj.el docs and Emacs Readme for chat, simplified setup 2023-03-27 22:02:47 +07:00
Debanjum
87090531da Install, Start and Configure Khoj Server from Emacs 
Merge pull request #193 from debanjum/simplify-khoj-server-setup-on-emacs

## Major Changes
- ae535a0 Configure Khoj chat using khoj.el by setting OpenAI API key in Emacs
- 82eb4bf Setup Khoj server on opening khoj.el
- 99d19dc Start Khoj server from Emacs using khoj.el
- c92d791 Install Khoj server from Emacs using khoj.el
  *This assumes you have python (<3.11) and pip installed in a system path*

### Sample Config
- Enable Khoj Chat by configuring you OpenAI API Key
- Specify Org Files, Directories to Index for Search (and Chat)
  By default, your org-agenda-files (include archive files)) are indexed
- Invoke khoj by calling `C-c s`

``` emacs-lisp
(use-package khoj
  :after org
  :straight (khoj
             :type git
             :host github
             :repo "debanjum/khoj"
             :files ("src/interface/emacs/khoj.el"))
  :bind ("C-c s" . 'khoj)
  :config (setq
           khoj-openai-api-key "<YOUR_OPENAI_API_KEY_FOR_KHOJ_CHAT>"
           khoj-org-directories '("~/docs/notes" "~/docs/journals")
           khoj-org-files '("~/docs/tasks.org" "~/docs/journal.org" "~/docs/archive.org")))
```
 2023-03-27 18:49:43 +07:00
Debanjum Singh Solanky
83a7ccd729 Fix docstrings and method ordering in khoj.el 2023-03-27 18:33:09 +07:00
Debanjum Singh Solanky
5c2327ee4f Configure org directories to index from khoj.el 
Converts paths to glob style regexes that will index all org files
recursively under the specified list of path

Should help setup for org-roam users from khoj.el
 2023-03-27 18:30:53 +07:00
Debanjum Singh Solanky
6e8a40906d Allow disabling automatic server setup. Fix server start vs ready logic 
- khoj-auto-setup controls whether to automatically check for and
  setup khoj server from within Emacs
- extract install, start, configure sequence into public, interactive
  method. Allows calling khoj-setup during package load via init.el

- Fix: Do not attempt to configure or wait for server ready if
  user has said no to auto-setup request
- Fix logic to mark server started vs ready
  - Previously the started/running vs ready variables defs were getting
    intertwined
  - Server started indicates server bootup has been triggered
  - Server ready indicates server API ready to accept requests
 2023-03-27 17:53:08 +07:00
Debanjum Singh Solanky
526a927bce Fix org entry extraction test, variable prefixed with khoj in khoj.el 
Discovered via failing build and test workflows on Github
 2023-03-27 16:44:50 +07:00
Debanjum Singh Solanky
7243059507 Track index update asynchronously via moon phase progressbar in khoj.el 2023-03-27 06:01:04 +07:00
Debanjum Singh Solanky
8a9055f918 Restrict server messages show in echo area to main server files 2023-03-27 04:59:55 +07:00
Debanjum Singh Solanky
ae535a06eb Configure Khoj chat using khoj.el by setting OpenAI API key in Emacs 2023-03-27 04:59:54 +07:00
Debanjum Singh Solanky
36b17d4ae0 Generalize the directory from config extraction elisp method 2023-03-27 03:44:03 +07:00
Debanjum Singh Solanky
924424c754 Throw actionable exceptions when content types or chat not configured 2023-03-27 02:47:44 +07:00
Debanjum Singh Solanky
359a2cacef Fix khoj--server-running to work with unconfigured or external server 
- If khoj server started outside emacs, khoj--server-ready should be set
to true by khoj--server-running method (instead of waiting for proc msg)

- If khoj server is unconfigured the /config/types endpoint wouldn't
return anything. Using config/data/default allows checking khoj server
running status without requiring it to be configured as well
 2023-03-27 02:45:59 +07:00
Debanjum Singh Solanky
d7fb9a596e Auto configure server before loading khoj-menu 
If the config hasn't changed there'll be no update. If config has
changed indexing will get triggered asynchronously. But user cannot
make query till indexing done

As easier to know when server ready to configure
 2023-03-27 02:44:02 +07:00
Debanjum Singh Solanky
8a21aff438 Make khoj.el server start, stop, restart, setup methods interactive 
No need to erase temporary buffers before working on them
 2023-03-27 01:53:15 +07:00
Debanjum Singh Solanky
cb40a96c85 Index configured org files from khoj.el 
- Set `khoj-org-files-index' to list of files to index
- Defaults to indexing org-agenda-files
- Uses khoj server api to configure org files to index
 2023-03-27 01:05:26 +07:00
Debanjum Singh Solanky
50760acc37 Wait for Khoj server to get ready before opening khoj.el transient menu 
- Use process filter, sentinel to mark when khoj server is ready or not
- Display server messages for visibility into server boot-up process
- Wait until server ready to open khoj transient menu in Emacs
  Until then khoj features wouldn't work anyway, so avoids confusion
 2023-03-26 13:00:01 +07:00
Debanjum Singh Solanky
82eb4bfd0d Setup Khoj server on opening khoj from with Emacs 
- Create helper methods to check, stop, restart, setup khoj server
- (Ask to) setup khoj server on calling khoj main entrypoint function
 2023-03-26 10:12:06 +07:00
Debanjum Singh Solanky
99d19dcf43 Start Khoj server from Emacs using khoj.el 2023-03-26 09:38:46 +07:00
Debanjum Singh Solanky
c92d79118a Install Khoj server from Emacs using khoj.el 2023-03-26 08:50:03 +07:00
Debanjum Singh Solanky
e281a498b4 Style Khoj search org buffer via elisp instead of in-buffer settings 2023-03-26 06:34:18 +07:00
Debanjum Singh Solanky
4f655d20ae Style Khoj chat directly via elisp instead of via in-buffer settings 2023-03-26 06:03:30 +07:00
Debanjum Singh Solanky
f6ff7b1beb Render foonote reference links as superscript for Khoj Chat on Emacs 2023-03-26 05:33:08 +07:00
Debanjum Singh Solanky
285a2b86d2 Use aiohttp version 3.8.4 as 4.x breaks docker image build 2023-03-26 05:33:02 +07:00
Debanjum Singh Solanky
67c850a4ac Add retry logic to OpenAI API queries to increase Chat tenacity 
- Move completion and chat_completion into helper methods under utils.py
- Add retry with exponential backoff on OpenAI exceptions using
  tenacity package. This is officially suggested and used by other
  popular GPT based libraries
 2023-03-26 05:12:35 +07:00
Debanjum
0aebf624fc Improve Khoj Chat in Emacs, Server 
Merge pull request #192 from debanjum/improvements-to-khoj-chat-in-emacs

### Khoj Chat on Emacs Improvements
- d78454d Load Khoj Chat buffer before asking for query to provide context
- 93e2aff Use org footnotes to add references, allows jump to def on click
- 5e9558d Stylize reference links as superscripts and show definition on hover
- bc71c19 Use `m` or `C-x m` in-buffer keybindings to send messages to Khoj

### Khoj Chat Server Improvements
- 27217a3 Time chat API sub-components for performance analysis
- 508b217 Update Chat API, Logs, Interfaces to store, use references as list
- d4b3866 Truncate message logs to below max supported prompt size by chat model
- cf28f10 Register separate timestamps for user query and response by Khoj Chat
 2023-03-25 05:49:27 +07:00
Debanjum Singh Solanky
ff846f05c5 Clean-up khoj.el based on linting helpers and manual review 2023-03-25 05:47:49 +07:00
Debanjum Singh Solanky
7e36f421f9 Truncate message logs to below max supported prompt size by model 
- Use tiktoken to count tokens for chat models
- Make conversation turns to add to prompt configurable via method
  argument to generate_chatml_messages_with_context method
 2023-03-25 05:13:56 +07:00
Debanjum Singh Solanky
4725416fbd Use shortcut keybindings in buffer to ease sending messages to Khoj 2023-03-25 05:06:01 +07:00
Debanjum Singh Solanky
508b2176b7 Update Chat API, Logs, Interfaces to store, use references as list 
- Remove the need to split by magic string in emacs and chat interfaces
- Move compiling references into string as context for GPT to GPT layer
- Update setup in tests to use new style of setting references
- Name first argument to converse as more appropriate "references"
 2023-03-24 22:10:11 +07:00
Debanjum Singh Solanky
b08745b541 Keep chat messages at 1 empty line visible distance in khoj.el 
- Clean redundant concat, format string
- Improve variable name to emojified sender
 2023-03-24 22:10:11 +07:00
Debanjum Singh Solanky
27217a330d Time chat API sub-components for performance analysis 
Time and the search query extraction, search and response generation
components
 2023-03-24 20:39:41 +07:00
Debanjum Singh Solanky
5e9558d39d Stylize references shown as footnote links in chat messages 
- Render references as superscript
- Show reference definitions on hover over reference links to ease access
- Truncate reference def shown on hover to 70 char
  - Add continuation suffix, ..., when reference definition truncated
 2023-03-24 20:38:05 +07:00
Debanjum Singh Solanky
cf28f104c7 Register separate timestamps for user query and response by Khoj Chat 2023-03-24 18:31:58 +07:00
Debanjum Singh Solanky
93e2aff786 Add references as org footnotes instead of links 2023-03-24 18:31:42 +07:00
Debanjum Singh Solanky
d78454d4ad Load Khoj Chat buffer before asking for query to provide context 2023-03-24 13:43:46 +07:00
Debanjum
4070d13a96 Create Khoj Chat Interface in Emacs 
Merge pull request #191 from debanjum/create-chat-interface-on-emacs

- Render conversation history in a read-only org-mode buffer for Khoj Chat
- Add `chat` as a transient action in the Khoj transient menu
- Style chat messages as org-mode entries
  - Put received date in property drawer and keep it hidden/folded by default
  - Add Khoj chat response as child entry of the users associated question org entry
    This allows folding back-n-forth between user and Khoj for easier viewing
  - Render source notes snippets used as references for response as org-mode links
    Hovering mouse on link or opening links shows reference note snippets used
 2023-03-22 16:32:40 -06:00
Debanjum Singh Solanky
863933daaa Resolve build issues found by melpazoid 2023-03-23 02:25:34 +04:00
Debanjum Singh Solanky
e9ca04af0d Require dash, org to run ERT tests for khoj.el 2023-03-23 01:46:26 +04:00
Debanjum Singh Solanky
06df394d6c Style chat messages as org-mode entries in Emacs 
- Style Message as Org Entries instead of List
- Put khoj response as child of user query entry
  - Improves color coding for readability
  - Allows folding each back-n-forth
- Put timestamp of message received into property drawer
- Use standardized time format for new and old chat messages
 2023-03-22 12:00:43 -06:00
Debanjum Singh Solanky
364e6c11af Render chat history from API in chat buffer on first run 
- Generalize the render-chat-response method to handle rendering
  history or chat response from chat API reponse

- Trigger rendering of khoj chat history if Khoj chat buffer not
  created for this session yet
 2023-03-22 12:00:35 -06:00
Debanjum Singh Solanky
36b52fdd0a Properly escape reference links before rendering 
- Use org-insert-link method to improve link rendering robustness
  Previous simple mechanism to crete org-links would result in links
  escaping out of formating. Use a user-facing org-mode method to
  remove/reduce probability of this

- Replace newlines with space to render reference notes as links
 2023-03-22 11:05:38 -06:00
Debanjum Singh Solanky
72f63a6ef7 Add basic chat interface for Khoj on Emacs 
- Query khoj chat API to get Khoj Chat response to user message
- Render chat messages as a org-mode list in format:
  - [sender-name]: *[message]*
    - /[receive-date]/
- Add references as org links with context visible on hover,
  but no jump to note
- Require dash library for khoj.el to simplify list manipulation.
  Use `-map-indexed' method from dash
 2023-03-22 10:47:55 -06:00
Debanjum Singh Solanky
e4d67694e1 Add search to method, variable names meant for khoj search in khoj.el 
In preparation to introduce Khoj chat in Emacs
 2023-03-21 21:44:11 -06:00
Debanjum Singh Solanky
98e5ea4940 Fix name of default encoder to replace in multi-lingual model setup docs 2023-03-21 20:38:17 -06:00
Debanjum Singh Solanky
2f6284872d Mention Khoj needs Python version 3.10 or lower in docs 2023-03-20 15:18:19 -06:00
Debanjum Singh Solanky
a9b81975f2 Fix encoder model name to configure multilingual search in Readme 
See comment in issue #98 for stale model name comment
 2023-03-19 17:27:53 -06:00
Debanjum
b351cfb8a0 Add Search Actor to Improve Querying Notes for Khoj Chat 
Merge pull request #189 from debanjum/add-search-actor-to-improve-notes-lookup-for-chat

### Introduce Search Actor
Search actor infers Search Queries from user's message
- Capabilities
  - Use previous messages to add context to current search queries[^1]
    This improves quality of responses in multi-turn conversations. 
  - Deconstruct users message into multiple search queries to lookup notes[^2]
  - Use relative date awareness to add date filters to search queries[^3]

- Chat Director now does the following:
  1. [*NEW*] Use Search Actor to generate search queries from user's message
  2. Retrieve relevant notes from Knowledge Base using the Search queries
  3. Pass retrieved relevant notes to Chat Actor to respond to user

### Add Chat Quality Tests 
- Test Search Actor capabilities
- Mark Chat Director Tests for Relative Date, Multiple Search Queries as Expected Pass

### Give More Search Results as Context to Chat Actor
- Loosen search results score threshold to work better for searches with date filters
- Pass more search results (up to 5 from 2) as context to Chat Actor to improve inference

[^1]: Multi-Turn Example
Q: "When did I go to Mars?"
Search: "When did I go to Mars?"
A: "You went to Mars in the future"
Q: "How was that experience?"
Search: "How my Mars experience?"
*This gives better context for the Chat actor to respond* 
[^2]: Deconstruct Example: 
Is Alpha older than Beta? => What is Alpha's age? & When was Beta born?

[^3]: Date Example: 
Convert user messages containing relative dates like last month, yesterday to date filters on specific dates like dt>="2023-03-01"
 2023-03-18 18:02:12 -06:00
Debanjum Singh Solanky
601ff2541b Revert to using GPT to extract search queries from users message 
- Reasons:
  - GPT can extract date aware search queries with date filters
    better than ChatGPT given the same prompt.
  - Need quality more than cost savings for now.
  - Need to figure ways to improve prompt for ChatGPT before using it
 2023-03-18 17:56:13 -06:00
Debanjum Singh Solanky
e28526bbc9 Extract search queries from users message using ChatGPT as Search Actor 
- Reasons
  - ChatGPT should be better at following instructions than GPT
  - At 1/10th the cost, it's much cheaper than using older GPT models
 2023-03-18 16:33:24 -06:00
Debanjum Singh Solanky
939d7731da Fix-up Search Actor GPT's response for decoding it as valid JSON 2023-03-18 16:30:55 -06:00
Debanjum Singh Solanky
f63fd0995e Pass more search results as context to Chat Actor to improve inference 2023-03-18 16:30:55 -06:00
Debanjum Singh Solanky
10836dedee Search should return user message if GPT response is not valid JSON 
Previously would throw if GPT response is not valid JSON. Better to
return original message to use for search instead
 2023-03-18 16:30:55 -06:00
Debanjum Singh Solanky
08f5fb315f Add answers to context for Search Actor to generate relevant queries 
Update Search Actor prompt with answers, more precise primer and
two more examples for context

Mark the 3 chat quality tests using answer as context to generate
queries as expected to pass. Verify that the 3 tests pass now, unlike
before when the Search Actor did not have the answers for context
 2023-03-18 16:30:55 -06:00
Debanjum Singh Solanky
f09bdd515b Expect Chat Director can extract relative dates using new Search Actor 2023-03-18 16:30:55 -06:00
Debanjum Singh Solanky
36c7389b46 Test Search Actor generating search query from Chat History 2023-03-18 16:30:55 -06:00
Debanjum Singh Solanky
2600cc9d4d Test Search Actor extracting relative dates & multiple questions 2023-03-18 16:30:55 -06:00
Debanjum Singh Solanky
45cb510421 Loosen search results score thresold used by chat for more context 2023-03-18 16:30:55 -06:00
Debanjum Singh Solanky
d871e04a81 Use past user messages, inferred questions as context to extract questions 
- Keep inferred questions in logs
- Improve prompt to GPT to try use past questions as context
- Pass past user message and inferred questions as context to help GPT
  extract complete questions
- This should improve search results quality

- Example Expected Inferred Questions from User Message using History:
  1. "What is the name of Arun's daughter?"
    => "What is the name of Arun's daughter"
  2. "Where does she study?" =>
    => "Where does Arun's daughter study?" OR
    => "Where does Arun's daughter, Reena study?"
 2023-03-18 16:30:50 -06:00
Debanjum Singh Solanky
1a5d1130f4 Generate search queries from message to answer users chat questions 
The Search Actor allows for
1. Looking up multiple pieces of information from the notes
   E.g "Is Bob older than Tom?" searches for age of Bob and Tom in 2 searches
2. Allow date aware user queries in Khoj chat
   Answer time range based questions
   Limit search to specified timeframe in question using date filter
   E.g "What national parks did I visit last year?" adds
   dt>="2022-01-01" dt<"2023-01-01" to Khoj search

Note: Temperature set to 0. Message to search queries should be deterministic
 2023-03-18 16:28:51 -06:00
Debanjum Singh Solanky
d0f14d3f85 Test usage of = in date filter queries 2023-03-16 14:52:59 -06:00
Debanjum Singh Solanky
dfb277ee37 Set skipif at module level if OpenAI API key not set for chat tests 
- Remove stale message_to_prompt test
  It is too broad, reduces maintainability.
  Remove as it doesn't really need its own test right now
- Setting skipif at module level for chat actor, director tests
  reduces code duplication as earlier was using decorator on each chat
  test
 2023-03-16 12:23:52 -06:00
Debanjum
e75e13d788 Create Tests to Measure Chat Quality, Capabilities 
Create Rubric to Test Chat Quality and Capabilities

### Issues
- Previously the improvements in quality of Khoj Chat on changes was uncertain
- Manual testing on my evolving set of notes was slow and didn't assess all expected, desired capabilities

### Fix
1. Create an Evaluation Dataset to assess Chat Capabilities
   - Create custom notes for a fictitious person (I'll publish a book with these soon 😅😋)
   - Add a few of Paul Graham's more personal essays. *[Easy to get as markdown](https://github.com/ofou/graham-essays)*
2. Write Unit Tests to Measure Chat Capabilities
   - Measure quality at 2 separate layers
     - **Chat Actor**: These are the narrow agents made of LLM + Prompt. E.g `summarize`, `converse` in `gpt.py`
     - **Chat Director**: This is the chat orchestration agent. It calls on required chat actors, search through user provided knowledge base (i.e notes, ledger, image) etc to respond appropriately to the users message.  This is what the `/api/chat` API exposes.
   - Mark desired but not currently available capabilities as expected to fail <br />
     This still allows measuring the chat capability score/percentage while only failing capability tests which were passing before on any changes to chat
 2023-03-16 11:30:52 -06:00
Debanjum Singh Solanky
4e15b4e411 Create test notes dataset for chat testing 
Combine hand-written custom notes and PG essays with personal
content to bulk up notes count

Delete old documentation markdown as not a representative dataset for
application (which is more tuned for personal notes)
 2023-03-16 09:30:37 -06:00
Debanjum Singh Solanky
1b4d562700 Test Chat Director Capabilities: Answer from notes, chat history etc 
- Chat directors are broad agents.
  - Chat directors orchestrate narrow actor agents to synthesize
    final response for the user
  - Agents are Prompts + ML Model

- Test Chat Director Capabilities
  1. [X] Answer from retrieved notes
  2. [X] Answer from chat history
  3. [X] Answer general questions
  4. [X] Carry out multi-turn conversation
  5. [X] Say don't know when answer not in provided context
  6. [X] Answers that require current date awareness
     This test is expected to fail as the chat is not capable of doing
     this without the Search actor. But the test allows assessing chat quality
  7. [X] Date-aware aggregation across multiple different notes
     This test is expected to fail as the chat is not capable of doing
     this without the Search actor. But the test allows assessing chat quality
  8. [X] Ask clarification questions if no unambiguous answer in provided context
  9. [X] Retrieve answer from chat history beyond lookback window
     This test is expected to fail as the chat director is not capable
     of searching chat history yet. But the test allows assessing chat quality
 10. [X] Retrieve context for answer using multiple independent
         searches on knowledge base
     This test is expected to fail as the chat is not capable of doing
     this without the Search actor. But the test allows assessing chat quality
 2023-03-16 09:30:37 -06:00
Debanjum Singh Solanky
b6d63137f1 Setup Pytest fixture for conversation processor to test chat API 
- Index markdown test data as knowledge base. As easier to get good
  markdown content (vs org)
- Setup markdown_content_config, processor_config and chat_client to
  test chat API
 2023-03-16 09:30:37 -06:00
Debanjum Singh Solanky
3f719c9e17 Rename Chat Model+Prompt tests to chat actor tests 2023-03-16 09:30:37 -06:00
Debanjum Singh Solanky
7526a50dd4 Extract conversation processor utility funcs from gpt.py into utils.py 2023-03-16 09:30:37 -06:00
Debanjum Singh Solanky
7c4d546039 Configure tests to mark chat quality tests & filter unhelpful warnings 
- Mark chat quality tests, register custom mark for chat quality
- Filter unhelpful deprecation warnings from within dateparser library
- Error if tests use unregistered marks
 2023-03-16 09:30:37 -06:00
Debanjum Singh Solanky
c1128a1ad8 Test Chat Actor Capabilities; ability to answer from notes, chat logs etc 
- Chat actors are narrow agents (prompt + ML model)
  Chat actors are different from the Chat director. who orchestrates
  the narrow actor agents to synthesize final response to the user

- Test Chat Actor Capabilities
  1. Answer from retrieved notes
  2. Answer from chat history
  3. Answer general questions
  4. Carry out multi-turn conversation
  5. Say don't know when answer not in provided context
  6. Answers that require current date awareness
  7. Date-aware aggregation across multiple different notes
  8. Ask clarification questions if no unambiguous answer in provided context
     This test is expected to fail as the chat is not capable of doing
     this consistently yet. But having the test allows assessing chat quality

- Use Openai API Key from OPENAI_API_KEY environment variable
- Gitignore .env file, python virtualenv directory
  Put OpenAI API Key in .env file to run chatbot tests via vscode
  The .env file is default location for importing env vars
 2023-03-16 09:30:37 -06:00
Debanjum Singh Solanky
9306cd901a Clean up chat tests to work with updated chat methods in gpt.py 2023-03-16 09:30:37 -06:00
Debanjum Singh Solanky
24ddebf3ce Make converse prompt more precise. Fix default arg vals in gpt methods 
- Set conversation_log arg default to dict
- Increase default temperature to 0.2 for a little creativity in
  answering
- Make GPT be more reliable in looking at past conversations for
  forming response
 2023-03-16 09:30:37 -06:00
Debanjum Singh Solanky
8609e3129e Fix, improve displaying chat messages, sources by Khoj in web interface 
Pretty pretty json in conversation logs
 2023-03-14 11:24:47 -06:00
Debanjum
6c0e82b2d6 Merge Improve Khoj Chat PR #183 from debanjum/improve-chat-interface 
# Improve Khoj Chat
## Main Changes
- Use the new [API](https://openai.com/blog/introducing-chatgpt-and-whisper-apis) for [ChatGPT](https://openai.com/blog/chatgpt) to improve conversation quality and cost
- Improve Prompt to answer query using indexed notes
  - Previously was asking GPT to summarize the notes
  - Both the chat and answer API use this new prompt
- Support Multi-Turn conversations
  - Pass previous messages and associated reference notes to ChatGPT for context
- Show note snippets referenced to generate response
  - Allows fact-checking, getting details
- Simplify chat interface by using only single unified chat type for now

## Miscellaneous
- Replace summarize with answer API. Summarize via API not useful for now
- Only pass Khoj search results above a threshold confidence to GPT for context
  - Allows Khoj to say don't know if it can't find answer to query from notes
  - Allows relying on (only) conversation history to generate response in multi-turn conversation
- Move Chat API out of beta. Update Readme
 2023-03-10 19:03:44 -06:00
Debanjum Singh Solanky
cccd225247 Deduplicate and simplify logic to render chat message with reference 2023-03-10 18:58:11 -06:00
Debanjum Singh Solanky
b9caad458e Type score_threshold with union, not |, to support python <3.10 2023-03-10 18:58:11 -06:00
Debanjum Singh Solanky
198d9af8cf Update Readme to reflect Khoj Chat out of Beta 2023-03-10 18:58:11 -06:00
Debanjum Singh Solanky
a71f168273 Move the chat API out of beta. Save chat sessions at 15min intervals 2023-03-10 17:20:52 -06:00
Debanjum Singh Solanky
bcc0bed9db Upgrade bump_version script to handle release and post-release commit 
- Updates version in khoj.el and Obsidian manifest, package, versions
  json files under interface and project root
- Create and tag release commit with updated files
- Creates commit with post-release version upgrade in files
- Use flags to specify whether to create a release or post-release commit
 2023-03-10 15:23:17 -06:00
Debanjum Singh Solanky
8bb8824d0c Bump khoj versions in obsidian, emacs files 2023-03-10 15:23:17 -06:00
Debanjum Singh Solanky
e16d0b6d7e Open references notes used for chat on mobile too (by clicking) 
Requires clicking the reference as hover doesn't work on mobile
 2023-03-09 17:13:07 -06:00
Debanjum Singh Solanky
c3c7b8a951 Make Khoj chat a separate Progressive Web App (PWA) for easier access 2023-03-09 13:45:06 -06:00
Debanjum Singh Solanky
3838f9d8e3 Remove explicitly asking GPT to say I don't know in prompt for now 
GPT still mostly says I don't know when answer not in notes or chats

But with this its more inclined to answer general questions not in
chats or notes while informing user that the information is not from
existing chats or notes
 2023-03-09 12:11:44 -06:00
Debanjum Singh Solanky
f7b8cdd02e Log prompts being passed to GPT for debugging 2023-03-08 19:17:52 -06:00
Debanjum Singh Solanky
2739a492b4 Log message metadata along with Khoj message instead of user message 
References should be attached to khoj chat messsage rather than the
users message in the chat interface
 2023-03-08 19:16:24 -06:00
Debanjum Singh Solanky
87d1e1341d Show reference notes used as response context in chat interface 2023-03-08 19:16:24 -06:00
Debanjum Singh Solanky
280061e1fa Do not deduplicate search results used for chat context 
- Chat uses compiled form of search results, not the raw entries to
  provide context for chat. The compiled snipped search results
  themselves are unique and using multiple of them for context from
  the same raw note is fine if they cross the score and rank thresholds

  This should improve the context provided for chat

- Also apply score_threshold, no deduplication to the answers API
 2023-03-06 23:51:31 -06:00
Debanjum Singh Solanky
672f61529e Make getting deduped search results configurable via Search API 2023-03-06 23:48:46 -06:00
Debanjum Singh Solanky
4fb628975c Fix jumping to note from Khoj Obsidian search modal result on Windows 
- Issue
  The file path separator by khoj server and the Obsidian vault were
  different on Windows
- Fix
  Normalize file path to use forward slash(/) to find the matching
  note file in the Obsidian vault for jump to it

Resolves #177
 2023-03-05 21:07:54 -06:00
Debanjum Singh Solanky
b6cdc5c7cb Do not expose answer API as a chat type in chat web interface or API 
Answer does not rely on past conversations, just the knowledge base.
It is meant for one off interactions, like search rather than a
continuing conversation like chat

For now it is only exposed via API. Later it will be expose in the
interfaces as well

Remove ability to select different chat types from the chat web
interface as there is only a single chat type

Stop appending answers to the conversation logs
 2023-03-05 18:21:59 -06:00
Debanjum Singh Solanky
7f994274bb Support multi-turn conversations in chat mode 
- Only use decent quality search results, if any, as context
- Pass source results used by previous chat messages as context
- Loosen prompt to allow looking at previous chats and notes to answer
- Pass current date for context

- Make GPT provide reason when it can't answer the question. Gives
  user context to tune their questions
 2023-03-05 18:21:39 -06:00
Debanjum Singh Solanky
d73042426d Support filtering for results above threshold score in search API 2023-03-05 18:21:39 -06:00
Debanjum Singh Solanky
45f461d175 Keep search results passed to GPT as context in conversation logs 
This will be useful to
1. Show source references used to arrive at answer
2. Carry out multi-turn conversations
 2023-03-05 16:00:19 -06:00
Debanjum Singh Solanky
7cad1c9428 Only use past chat message, not session summaries as chat context 
Passing only chat messages for current active, and summaries
for past session isn't currently as useful
 2023-03-05 16:00:18 -06:00
Debanjum Singh Solanky
ad1f1cf620 Improve and simplify Khoj Chat using ChatGPT 
- Set context by either including last 2 chat messages from active
  session or past 2 conversation summaries from conversation logs

- Set personality in system message
- Place personality system message before last completed back & forth
  This may stop ChatGPT forgetting its personality as conversation progresses given:
  - The conditioning based on system role messages is light
  - If system message is too far back in conversation history, the
    model may forget its personality conditioning
  - If system message at end of conversation, the model can think its
    the start of a new conversation
  - Inserting the system message before last completed back & forth should
    prevent ChatGPT from assuming its the start of a new conversation
    while not losing personality conditioning from the system message

- Simplfy the Khoj Chat API to for now just answer from users notes
  instead of trying to infer other potential interaction types.
  - This is the default expected behavior from the feature anyway
  - Use the compiled text of the top 2 search results for context

- Benefits of using ChatGPT
  - Better model
  - 1/10th the price
  - No hand rolled prompt required to make GPT provide more chatty,
    assistant type responses
 2023-03-05 01:24:13 -06:00
Debanjum Singh Solanky
9d42b5d60d Use multiple compiled search results for more relevant context to GPT 
Increase temperature to allow GPT to collect answer across multiple
notes
 2023-03-05 01:24:13 -06:00
Debanjum Singh Solanky
c3b624e351 Introduce improved answer API and prompt. Use by default in chat web interface 
- Improve GPT prompt
  - Make GPT answer users query based on provided notes instead
    of summarizing the provided notes
  - Make GPT be truthful using prompt and reduced temperature
  - Use Official OpenAI Q&A prompt from cookbook as starting reference
- Replace summarize API with the improved answer API endpoint
- Default to answer type in chat web interface. The chat type is not
  fit for default consumption yet
 2023-03-05 01:24:13 -06:00
Debanjum Singh Solanky
7184508784 Mention Python and Pip need to be installed in Main and Emacs Readme 2023-03-02 21:28:54 -06:00
Debanjum Singh Solanky
211e460398 Output date filter from cache log at debug level. Remove unused imports 
Other logs not directly useful to user have already been converted
to debug log levels in 1ae4016. Just forgot to convert this log line too
 2023-03-02 15:41:32 -06:00
Debanjum Singh Solanky
c823f46d89 Test error on missing fields in ContentConfig pulled from Khoj.yml 
Resolves #9
 2023-03-02 15:35:39 -06:00
Debanjum Singh Solanky
b6dbe4dd1d Do not try retrieve an unconfigured core content type in Config GUI 
Previous behavior was resulting in a null reference error. As key for
the core content/search type was not present in current config

Fallback to using default config for unconfigured core content type
instead

See #165 for details
 2023-03-02 11:09:31 -06:00
Debanjum Singh Solanky
1ae40163a9 Show user friendly information logs by default for context 
- Use emojis to make info logs easier to read
- Inform when khoj is ready to use
- Provide information on what khoj is doing while starting up
- Inform when content/search types and processors are setup
- Inform when models are being loaded from the web as this step can
  take time
- Convert all other info logs to be only shown in verbose mode
 2023-03-01 16:39:07 -06:00
Debanjum Singh Solanky
fe03ba3dce Index intro text before headings in org files 
- Text before headings was not being indexed due to buggy orgnode
  parsing logic
- Resolved indexing intro text from files with and without headings in
  them
- Ensure intro text node has heading set to all title lines collected
  from the file

Resolves #165
 2023-03-01 12:11:33 -06:00
Debanjum Singh Solanky
ed177db2be Emojify step names in workflows. Stop publishing to TestPyPi from PR 2023-03-01 10:56:39 -06:00
Debanjum Singh Solanky
7ad251b8ef Log and Continue on OSError while collating dates for date filters 
Log to understand if error, date can be handled better
Mitigates #172
 2023-03-01 01:23:37 -06:00
Debanjum Singh Solanky
2bed4c3b50 Fix configuring search types & /config/types API when no plugin configured 
- Test /config/types API when no plugin configured, only plugin configured
  and no content configured scenarios
- Do not throw null reference exception while configuring search types
  when no plugin configured
- Do not throw null reference exception on calling /config/types API
  when no plugin configured

Resolves bug introduced by #173
 2023-03-01 01:23:37 -06:00
Debanjum Singh Solanky
8914dbd073 Fix creating GUI panels for unconfigured search, processor types 
Repro:
1. Open khoj server with `khoj` on first run
2. Install/enable Khoj Obsidian plugin (to configure khoj server)
3. Restart khoj server with `khoj`

Bug:
- Unconfigured processor and search_types are instantiated as None in
  self.current_config
- While creating the desktop GUI, these null configs are attempted to
  be accessed as valid dictionaries for creating their GUI panels
- This results in the null ref errors

Fix:
Use default config to create their GUI elements for unconfigured
search and processor types

Resolves #167
 2023-03-01 01:20:58 -06:00
Debanjum
e77a5ffc83 Merge pull request #173 from debanjum/enable-creating-content-plugins 
## Enable Creating Content Plugins

### Goal
Index, Search text content not supported by default in Khoj using plugins

### Code Changes
- fcbbe8c Configure content plugins to index using `khoj.yml`
- Index content plugins from standardized JSONL format for ingestion
  - 55a032e Add jsonl processor to index plugin content
  - ab0d3a0 Index configured plugins on app start and via update API endpoint
- Expose plugin content types for usage by interfaces
  - 47b58a2 Dynamically update available types on loading the Khoj server
  - Expose indexed types via API (9d38ead). Simplify getting enabled types in Web (f3f2438), Emacs (1e43f1a) interfaces
- Search plugin content from the Web and Emacs Interfaces
  - d91c7e2 Search plugin content via the search API
  - Render plugin content on Web (88344f9) and Emacs (c2814fc) interfaces
    - The Web, Emacs interfaces are general interfaces, they allow searching across all content types
    - The Obsidian interface is currently tuned for only markdown content
      It will be extended to render more content plugins later

### Testing
- fcbbe8c Add unit tests to test reading plugin config from khoj.yml
- 55a032e Add unit tests for the `JsonlToJsonl` processor
- 88a9ead Add unit tests to validate search, incremental update, force-update API works with plugin content types
- b09350c Add unit test to validate only configure search types returned by the new /api/config/types API endpoint
- Manually test the config read, indexing, search and update with local khoj
 2023-02-28 22:23:25 -06:00
Debanjum Singh Solanky
b09350c052 Fix to return only enabled content types via the new config/types API 
- Previously was return all core content types even if they had not been
  setup
- Add test to validate only configured content types are returned by
  the api/config/types API endpoint
 2023-02-28 22:08:26 -06:00
Debanjum Singh Solanky
b177adf3a7 Return value of search_type in /config/type API endpoint 
- Remove need for interfaces to downcase content types returned by API
  before using the type in search and other API endpoint
- Fix to check for search_type.name in plugin keys instead of value
 2023-02-28 21:49:26 -06:00
Debanjum Singh Solanky
ede6eb6879 Re-enable testing search and update API with image content type 
It may have been disabled due to issues with image search earlier
 2023-02-28 20:25:51 -06:00
Debanjum Singh Solanky
88a9eadfba Use client pytest fixture to test API with plugin type configured 2023-02-28 20:25:51 -06:00
Debanjum Singh Solanky
ab501a56c9 Create pytest fixture to configure app with plugin, search types 2023-02-28 20:25:51 -06:00
Debanjum Singh Solanky
f944408e69 Update content_config pytest fixture to index plugin content 2023-02-28 20:25:51 -06:00
Debanjum Singh Solanky
88344f9ed2 Improve rendering search results of plugin content types on web interface 
Render only the entry from plugin search response instead of raw json
Use the results-ledger styling for results-plugin styling
 2023-02-28 20:25:51 -06:00
Debanjum Singh Solanky
c2814fce58 Improve rendering search results of plugin content types in khoj.el 
Render only the entry from plugin search response instead of raw json
 2023-02-28 20:25:51 -06:00
Debanjum Singh Solanky
f3f24387ec Use new config/types API to set enabled content types on web interface 2023-02-28 20:25:51 -06:00
Debanjum Singh Solanky
1e43f1a12e Use new config/types API to set enabled content types in khoj.el menu 2023-02-28 20:25:51 -06:00
Debanjum Singh Solanky
9d38eadd42 Return enabled content types via api/config/types API endpoint 
Simplifies dynamically populating enabled content types for interfaces
 2023-02-28 20:25:51 -06:00
Debanjum Singh Solanky
68bd5d9ebc Configure API routes after set up search types while configuring server 
Configure app routes after configuring server.
Import API routers after search type is dynamically populated.
Allow API to recognize the dynamically populated plugin search types
as valid type query param.
Enable searching for plugin type content.
 2023-02-28 20:25:51 -06:00
Debanjum Singh Solanky
d91c7e2761 Search for plugin content via the search API 2023-02-28 20:25:51 -06:00
Debanjum Singh Solanky
47b58a2a4d Configure, use dynamically instantiated SearchType enum on app start 
The SearchType is now dynamically populated with core and configured
plugin types

Use the new dynamic SearchType enum from state.py across codebase
 2023-02-28 20:25:51 -06:00
Debanjum Singh Solanky
ab0d3a08e2 Index configured plugins on app start and via update API endpoint 2023-02-28 20:25:51 -06:00
Debanjum Singh Solanky
55a032e8c4 Add processor to index entries from jsonl files for plugins 
- Read, merge entries from input jsonl files and filters
- Mark new, modified entries for update
 2023-02-24 02:54:12 -06:00
Debanjum Singh Solanky
fcbbe8c759 Read content plugin configs from Khoj config YAML 
Configure external text content plugins via the Khoj YAML
Reuse existing TextContentConfig definition for external text content plugins
 2023-02-23 23:57:32 -06:00
Debanjum Singh Solanky
f57d7bf5ad Use pypi khoj to fix docker builds and dockerize github workflow 
- Instead of building the package locally like before
  The issue started since moving to dynamic git based versioning with hatch-vcs
  This should reduce image size of docker builds too

- Also move to ubuntu image since pyqt6 builds available on it, so do
  not need to build it locally for image

- This s
 2023-02-19 01:57:01 -06:00
Debanjum Singh Solanky
fada617faa Fix TOC links, Add how to auto start Khoj server to Readme 
Rename tools directory to more standard scripts directory
 2023-02-18 23:51:02 -06:00
Debanjum Singh Solanky
61b6ee2857 Use helper script to bump khoj pre-release versions 2023-02-17 20:31:51 -06:00
Debanjum Singh Solanky
47c2cc63e1 Automate uploading Obsidian artifacts to new releases 2023-02-17 19:57:44 -06:00
Debanjum Singh Solanky
a8940462c4 Automate khoj python package versioning using hatch-vcs and Git tags 2023-02-17 18:19:01 -06:00
Debanjum Singh Solanky
053d6141f3 Ignore ts typing error, Fix SPDX license identifier in Obsidian plugin 2023-02-17 18:19:01 -06:00
Debanjum Singh Solanky
47569da38e Fix usage of "\" in orgnode test string to resolve DeprecationWarning 2023-02-17 17:15:44 -06:00
Debanjum Singh Solanky
36be3c4b8f Fix or ignore MyPy issues in PyQt desktop GUI code 
- Remove unneeded type ignore for mps with the latest mypy
- Stop excluding PyQT desktop GUI code from MyPy checks
- Do not warn about unused ignores. Some issue with mypy giving
  different errors in different environments (venv, system and pre-commit)
 2023-02-17 16:13:05 -06:00
Debanjum Singh Solanky
fd0a2f55f8 Run mypy checks in test workflow and on push (via pre-commit) 
- Run mypy on git push (not every commit) but for all files
  - Running it on pre-commit, doesn't make sense as mypy wants to look
    at all files, not just diff files
  - But this is too time consuming to run every commit, so run on push

- Update development section documentation on installing, manually
  running pre-commit for validation that includes running mypy checks
 2023-02-17 16:08:56 -06:00
Debanjum Singh Solanky
5c0d340970 Update Development section in Readme. Add steps for code validation 2023-02-17 13:31:37 -06:00
Debanjum Singh Solanky
051f0e3fb5 Add, configure and run pre-commit locally and in test workflow 2023-02-17 13:31:36 -06:00
Debanjum Singh Solanky
5e83baab21 Use Black to format Khoj server code and tests 2023-02-17 11:55:17 -06:00
Debanjum Singh Solanky
6130fddf45 Install pytest as optional dev dependency of app in test workflow 2023-02-17 10:11:57 -06:00
Debanjum Singh Solanky
8b293edd7c Move mypy config into pyproject.toml. Ignore 2 remaining mypy issues 2023-02-16 03:33:08 -06:00
Debanjum Singh Solanky
7a9a811874 Fix authors, homepage URL in pyproject.toml and workflow triggers 2023-02-16 03:19:56 -06:00
Debanjum Singh Solanky
dcb86c2d3e Build khoj python package using hatchling, pyproject.toml 
- Why
  - pyprojects.toml is the python standards compliant config format
    - allows collating python tooling configs into single standard file
  - hatch(-ling) is a new lightweight build system for python packages

- Detailed Changes
  - Replace setup.py, setuptools with pyproject.toml, hatchling for
    khoj python config and build
  - move pytest into optional development dependencies
  - add more links to khoj in the project urls section
  - add topic classifiers and keywords to find khoj package

  - Delete setup.py, MANIFEST.in as moved to pyproject.toml based setup
  - Update pypi workflow to set python package version in pyproject.toml
 2023-02-16 02:37:32 -06:00
Debanjum Singh Solanky
c641eb4ad6 Improve rendering log and error stacktraces using the Rich package 
- Use Rich to render uvicorn, fastAPI logs as well
  The previous CustomFormatter only worked on khoj logs
- Improve rendering stacktrace on errors using Rich
 2023-02-15 16:19:32 -06:00
Debanjum Singh Solanky
a403def19e Fix workflow to publish Khoj python package to PyPi 2023-02-14 22:19:21 -06:00
Debanjum
eee57599ad Improve Dockerize, Publish to PyPi Workflows 
- fb86dea Create tagged Docker image on new tag/release
- 01fd98b Improve workflow to publish khoj to pypi
 2023-02-14 21:11:56 -06:00
Debanjum Singh Solanky
af6d65a909 Create tagged Docker image on new tag/release 2023-02-14 20:04:06 -06:00
Debanjum Singh Solanky
25e06f26c0 Improve workflow to publish khoj to pypi 
- Use emoji's to improve visual indicator of action step
- Rename to pypi instead of the more ambiguous publish name
  Publish could mean publish docker image, publish to pypi, MELPA or
  Obsidian plugin
- Update workflow badge, link pypi badge to khoj pypi package page
- Use pypa official github action to upload package to (test) pypi
  instead of doing it manually using twine
- Upload python package artifact for easier access for testing.
  As uploading to testpypi doesn't work for PRs by others from forked repos
 2023-02-14 20:03:35 -06:00
Debanjum
11873795a6 Use src layout to fix packaging khoj for pypi 
### Issue
The khoj python package was using a common top level name[1], `src' instead of `khoj' due to incorrect usage of the src layout[2]

### Fix
Put content meant for python packaging from `src/' to `src/khoj/'
Update code, tests, configs and docs to reference new layout

The `khoj' python package should now get unpacked under `khoj' instead of `src' directory

### Details
- 25a749c Use the src/ layout to fix packaging Khoj for PyPi
- bc7477e Move Emacs, Obsidian plugin code out from under src/khoj directory
- f83cf4e Check wheel contents in workflow before publishing Khoj to PyPI

[1]: https://github.com/jwodder/check-wheel-contents#w005--wheel-contains-common-toplevel-name-in-library
[2]: https://packaging.python.org/en/latest/discussions/src-layout-vs-flat-layout/
 2023-02-14 16:26:07 -06:00
Debanjum Singh Solanky
e76c285bdc No need to prune plugins as not included in pypi package. 
Mention Obsidian as supported Interfaces in Readme
 2023-02-14 16:15:40 -06:00
Debanjum Singh Solanky
bc7477ea3e Move Emacs, Obsidian plugin code out from under src/khoj directory 
- What
  - The Emacs and Obsidian interfaces stay in their original
    directories under src/
  - src/khoj now only contains code meant for pypi packaging

- Benefits
  - This avoids having to update khoj MELPA, Obsidian plugin config as
    the Emacs, Obsidian code is under their original directories
  - It separates the code in src/khoj meant for python packaging from
    code for external interfaces like Emacs and Obsidian
 2023-02-14 15:44:22 -06:00
Debanjum Singh Solanky
f83cf4ebc6 Check wheel contents in workflow before publishing it to PyPI 2023-02-14 15:20:44 -06:00
Debanjum Singh Solanky
25a749ca1d Use the src/ layout to fix packaging Khoj for PyPi 
- Why
  The khoj pypi packages should be installed in `khoj' directory.
  Previously it was being installed into `src' directory, which is a
  generic top level directory name that is discouraged from being used

- Changes
 - move src/* to src/khoj/*
 - update `setup.py' to `find_packages' in `src' instead of project root
 - rename imports to form `from khoj.*' in complete project
 - update `constants.web_directory' path to use `khoj' directory
 - rename root logger to `khoj' in `main.py'
 - fix image_search tests to use the newly rename `khoj' logger
 - update config, docs, workflows to reference new path `src/khoj'
 2023-02-14 15:19:06 -06:00
Debanjum Singh Solanky
cc31cd070d Enable the publish workflow for PRs created in the main repo 
The publish workflow was previously disabled for PRs in commit
d1945c5ba8
 2023-02-14 13:51:31 -06:00
Debanjum
84322b2a45 Demo using Search in Khoj Obsidian Plugin 2023-02-14 08:43:50 -08:00
Debanjum Singh Solanky
a4dcb20622 Add setting to toggle auto configuring of khoj backend from Obsidian 
- By default the obsidian plugin automatically configures the khoj
  backend to index the current vault
- For more complex scenarios, users can manage their ~/.khoj/khoj.yml
  manually by toggling the auto-configure setting off in the khoj
  plugin settings

Resolves #156
 2023-02-13 20:15:28 -06:00
Debanjum Singh Solanky
24aa696ef5 Indicate indexing active on Update button in Obsidian plugin settings 
Use moon rotating through phases to indicate notes indexing in progress

Resolves #129
 2023-02-13 19:28:19 -06:00
Debanjum Singh Solanky
11517ba8eb Encode jsonl data as utf8 for gzip write for consistent read/write encoding 
Should help with issue #89
 2023-02-12 17:33:23 -06:00
Debanjum Singh Solanky
c156b3e087 Remove sub-dependencies from setup.py. Upgrade sentence-transformer 
- setup.py best practise recommends only specifying core dependencies,
  not dependencies of core dependencies in it

- Latest sentence-transformer (version 2.2.2) correctly installs its
  huggingface_hub dependency. Else application fails to start
 2023-02-12 10:42:05 -06:00
Debanjum Singh Solanky
3ec41c4d64 Wrap lines for org, markdown results in khoj search results buffer 2023-02-12 07:33:50 -06:00
Debanjum Singh Solanky
d1945c5ba8 Do not run publish workflow for PRs as forks do not have auth token 2023-02-12 07:31:24 -06:00
Debanjum Singh Solanky
9a013ec48f Add more details to setup Khoj backend in Obsidian plugin readme 2023-02-12 07:31:13 -06:00
Debanjum
24c553877c Merge pull request #152 from axelson/fix-obsidian-doc-link 
Fix link to Obsidian plugins doc in Khoj Obsidian Readme
 2023-02-10 22:20:06 -06:00
Jason Axelson
6d5930363a Fix obsidian plugins doc link 
Also make it more obvious where the link is going, initially I thought
the link was to another official khoj documentation site.
 2023-02-10 07:11:21 -10:00
Debanjum Singh Solanky
215235efd2 Bump khoj pre-release version 2023-02-08 20:24:36 -03:00
Debanjum Singh Solanky
55e4fa9719 Fix indentation in workflow yaml for testing khoj backend 2023-02-07 02:59:46 -03:00
Debanjum Singh Solanky
2445664d40 Deprioritize searching for Music content over other text content 2023-02-07 02:41:31 -03:00
Debanjum Singh Solanky
2e052913b6 Search in first configured content type when no search type set 
Instead of searching through all configured content types but only
returning results of the last configured content type
 2023-02-07 02:41:31 -03:00
Debanjum Singh Solanky
a26ab31d20 Allow chat with markdown notes if no org-mode content configured 2023-02-07 02:41:31 -03:00
Debanjum
99a03da3f7 Read Markdown file as utf8 instead of the default encoding used by OS 
### Background
  1. Obsidian stores markdown notes as `utf8`[1]
  2. By default, the python `open` command uses the OS locale encoding[2]

### Issue
  Based on above background, if the OS locale encoding isn't `utf8` it causes the `UnicodeDecodeError: <locale_encoding> codec can't decode byte` error

### Fix
  - Read markdown files as `utf8`
    The Obsidian plugin is the main use-case for markdown files in khoj currently and that stores md files as `utf8`.
    Do not assume utf8 for other content types like org-mode, beancount for now.
  - Fail if error in reading file as utf8, instead of ignoring errors.
    Would rather have user realize that their files are not going to get indexed correctly.

[1]: https://forum.obsidian.md/t/better-handle-md-files-not-stored-in-utf8-format/13524/3
[2]: https://docs.python.org/3/library/functions.html#open
 2023-02-07 01:46:42 -03:00
Debanjum Singh Solanky
d3e82b918f Make Khoj require python version below 3.11 until PyTorch works with it 
Closes #128
 2023-02-06 23:11:51 -03:00
Debanjum Singh Solanky
c11f7b47e4 Update workflow to run backend tests for all supported python versions 2023-02-06 21:05:34 -03:00
Debanjum Singh Solanky
11a18cc452 Update khoj docker config to index sub directories for text content 
- Khoj supports indexing subdirectories but the khoj docker config
  wasn't updated to support the same
- This should also allow khoj docker users to index multiple separate
  directory trees by mounting them into separate sub folders within
  /data/<content-type>/.
  For e.g /data/org/dir1, /data/org/dir2 etc in khoj_docker.yml
 2023-02-06 21:04:50 -03:00
Debanjum Singh Solanky
fbb7747dcc Read Markdown file as utf8 instead of the default encoding used by OS 
- Background
  1. Obsidian stores markdown notes as utf8[1]
  2. By default, the python `open' command uses the OS locale encoding[2]

  This was causing the `UnicodeDecodeError: <locale_encoding> codec can't decode byte' error

- Fix
  - Read markdown files as utf8
    The Obsidian plugin is the main use-case for markdown files in
    khoj currently and that stores md files as utf8.
    Do not assume utf8 for other content types like org-mode, beancount for now.
  - Fail if error in reading file as utf8, instead of ignoring errors.
    Would rather have user realize that their files are not going to
    get indexed correctly.

[1]: https://forum.obsidian.md/t/better-handle-md-files-not-stored-in-utf8-format/13524/3
[2]: https://docs.python.org/3/library/functions.html#open
 2023-02-06 21:04:50 -03:00
Debanjum Singh Solanky
66dca6cf33 Add Docs to Search across Languages, Uninstall Khoj to Readme 
Add details and fixes to Obsidian, Main readme
based on feedback, confusion from the Obsidian plugin announcement
 2023-02-06 21:04:50 -03:00
Debanjum Singh Solanky
cba9a6a703 Use List, Tuple, Set from typing to support Python 3.8 for khoj 
Before Python 3.9, you can't directly use list, tuple, set etc for
type hinting

Resolves #130
 2023-02-06 01:23:52 -03:00
Debanjum
14f28e3a03 Mention Emacs, Obsidian plugins at top of main Readme 
Add badges for supported plugins at top of main readme.
Link badges to plugin docs for easy navigation for plugin users from main readme/project root
 2023-01-28 18:01:20 -08:00
Debanjum Singh Solanky
f26cee604d Update Khoj Plugin Install Instructions. Rename main Readme to README 
Khoj plugin page from within Obsidian isn't recognized. Seems like it
needs an uppercase readme file only. So it doesn't show the Khoj
readme from within Obsidian itself.
 2023-01-27 20:01:31 -03:00
Debanjum Singh Solanky
2e13e15625 Ensure markdown entries in khoj.el results separated by empty line 
- Update khoj.el test to reflect updated rendering logic
- Move ledger render function before image rendered to group functions
  with similar logic closer
 2023-01-26 19:13:02 -03:00
Debanjum Singh Solanky
85ae46f429 Use thread_last to make results rendering funcs more readable in khoj.el 2023-01-26 18:59:44 -03:00
Debanjum
a8ab9448da Resolve Khoj Obsidian Plugin feedback 
### Details
- b415f87 Split find and jump to notes code in `onChooseSuggestion' method
- 37063f6 Truncate query to 8k chars for find similar notes from Obsidian plugin
- 4456cf5 No need to use `then' or `finally' in `async' functions after an `await'
- 4070be6 Pass app object from plugin instance to child objects and functions
- c203c6a Use Sentence case for Find similar mote Obsidian command name
 2023-01-26 18:54:33 -03:00
Debanjum Singh Solanky
b415f87093 Split code in onChooseSuggestion method to make it more readable 
Split find file, jump to file code to make onChooseSuggestion more readable
- Use find, instead of using return in forEach to get first match
- Move the jump to file+heading code out from forEach
 2023-01-26 18:26:24 -03:00
Debanjum Singh Solanky
37063f6a38 Truncate query to 8k chars for find similar notes from obsidian plugin 
Truncate current file data passed to khoj backend API via query string
below default query size supported by popular servers
 2023-01-26 18:26:24 -03:00
Debanjum Singh Solanky
4456cf5c8f No need to use then or finally in async functions after an await 2023-01-26 18:26:24 -03:00
Debanjum Singh Solanky
4070be637c Pass app object from plugin instance to child objects and functions 
Do not reference global app object from child objects and funcs
directly.

It is only available for debugging purposes and access to it maybe
dropped in the future.
 2023-01-26 18:26:24 -03:00
Debanjum Singh Solanky
c203c6a3fd Use Sentence case for Find Similar Note command name in Khoj Obsidian 2023-01-26 18:26:24 -03:00
Debanjum Singh Solanky
e18124ef6f Add badge for tests and update project subtitle in khoj.el Readme 2023-01-23 20:52:03 -03:00
Debanjum
477ef28e08 Create and Automate Tests for Khoj.el on Emacs 
- Use ERT to test `khoj.el'
- Test extracting and rendering of Org, Markdown and Ledger entries from Khoj API response
- Automate `khoj.el' testing using Github workflow
- Fix, Simplify and Test the get text around point code for the "Find Similar" feature
 2023-01-23 20:40:18 -03:00
Debanjum Singh Solanky
f9fb58aec3 Automate khoj.el testing using Github workflow 
Install transient.el dependency as it is not available by default
before Emacs 28.1
 2023-01-23 20:33:47 -03:00
Debanjum Singh Solanky
86e808abfb Test get-current-text helpers for Find Similar feature in khoj.el 2023-01-23 20:33:47 -03:00
Debanjum Singh Solanky
be6acda212 Create khoj.el tests. Test rendering results of each content types 2023-01-23 20:33:47 -03:00
Debanjum Singh Solanky
0d0bf3b5aa Simplify get-current-text functions for Find Similar in khoj.el 
Use existing functions like `string-trim', `thing-at-point' and
remove unneeded code from the two functions
 2023-01-23 19:15:52 -03:00
Debanjum Singh Solanky
07e9e4ecc3 Get current paragraph text when point at start of paragraph in khoj.el 
Previously if cursor was at start of current paragraph, it would get
text for the current and next paragraph, instead of just the current one
 2023-01-23 18:05:54 -03:00
Debanjum Singh Solanky
a0b03c8bb1 Get current entry text when point at heading for Find Similar in khoj.el 
Previously if cursor was at heading of current entry, it would find entries
similar to the previous outline heading, instead of the current one
 2023-01-23 10:01:25 -03:00
Debanjum Singh Solanky
013c7c10a4 Bump khoj pre-release version 2023-01-22 18:45:56 -03:00
200 changed files with 18823 additions and 5368 deletions
Expand all files Collapse all files

1
.dockerignore
View File

@@ -6,4 +6,5 @@ docs/
tests/
build/
dist/
scripts/
*.egg-info/

6
.github/workflows/build_khoj_el.yml vendored
View File

@@ -24,16 +24,16 @@ jobs:
- name: Set up Python 3.9
uses: actions/setup-python@v1
with: { python-version: 3.9 }
- name: Install
- name: ⏬️ Install Dependencies
run: |
python -m pip install --upgrade pip
sudo apt-get install emacs && emacs --version
git clone https://github.com/riscy/melpazoid.git ~/melpazoid
pip install ~/melpazoid
- name: Run
- name: 🌡️ Validate Khoj.el
env:
# Khoj recipe from https://github.com/melpa/melpa/pull/8321/files
RECIPE: (khoj :fetcher github :repo "debanjum/khoj" :files ("src/interface/emacs/*.el"))
RECIPE: (khoj :fetcher github :repo "khoj-ai/khoj" :files ("src/interface/emacs/*.el"))
EXIST_OK: true
LOCAL_REPO: ${{ github.workspace }}
run: echo $GITHUB_REF && make -C ~/melpazoid

20
.github/workflows/build.yml → .github/workflows/dockerize.yml vendored
View File

@@ -1,16 +1,18 @@
name: build
name: dockerize
on:
push:
tags:
- "*"
branches:
- master
paths:
- src/**
- src/khoj/**
- config/**
- setup.py
- pyproject.toml
- Dockerfile
- docker-compose.yml
- .github/workflows/build.yml
- .github/workflows/dockerize.yml
workflow_dispatch:
env:
@@ -22,19 +24,19 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Checkout Code
uses: actions/checkout@v2
uses: actions/checkout@v3
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v1
uses: docker/setup-buildx-action@v2
- name: Login to GitHub Container Registry
uses: docker/login-action@v1
uses: docker/login-action@v2
with:
registry: ghcr.io
username: ${{ github.repository_owner }}
password: ${{ secrets.PAT }}
- name: Build and Push Docker Image
- name: 📦 Build and Push Docker Image
uses: docker/build-push-action@v2
with:
context: .
@@ -42,4 +44,4 @@ jobs:
push: true
tags: ghcr.io/${{ github.repository }}:${{ env.DOCKER_IMAGE_TAG }}
build-args: |
PORT=8000
PORT=8000

47
.github/workflows/dockerize_telemetry_server.yml vendored Normal file
View File

@@ -0,0 +1,47 @@
name: dockerize telemetry server
on:
push:
branches:
- master
paths:
- src/telemetry/**
- .github/workflows/dockerize_telemetry_server.yml
pull_request:
branches:
- master
paths:
- src/telemetry/**
- .github/workflows/dockerize_telemetry_server.yml
workflow_dispatch:
env:
DOCKER_IMAGE_TAG: ${{ github.ref == 'refs/heads/master' && 'latest' || github.event.pull_request.number }}
jobs:
build:
name: Build Docker Image, Push to Container Registry
runs-on: ubuntu-latest
steps:
- name: Checkout Code
uses: actions/checkout@v3
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2
- name: Login to GitHub Container Registry
uses: docker/login-action@v2
with:
registry: ghcr.io
username: ${{ github.repository_owner }}
password: ${{ secrets.PAT }}
- name: 📦 Build and Push Docker Image
uses: docker/build-push-action@v2
with:
context: src/telemetry
file: src/telemetry/Dockerfile
push: true
tags: ghcr.io/${{ github.repository }}-telemetry:${{ env.DOCKER_IMAGE_TAG }}
secrets: |
"POSTHOG_API_KEY=${{ secrets.POSTHOG_API_KEY }}"

95
.github/workflows/publish.yml vendored
View File

@@ -1,95 +0,0 @@
name: publish
on:
push:
tags:
- "*"
branches:
- 'master'
paths:
- src/**
- setup.py
- .github/workflows/publish.yml
pull_request:
branches:
- 'master'
paths:
- src/**
- setup.py
- .github/workflows/publish.yml
jobs:
publish:
name: Publish App to PyPI
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up Python 3.10
uses: actions/setup-python@v4
with:
python-version: '3.10'
- name: Install Dependencies
run: |
python -m pip install --upgrade pip
pip install build twine
- name: Install Application
run: |
pip install --upgrade .
- name: Publish Release to PyPI
if: startsWith(github.ref, 'refs/tags')
env:
TWINE_USERNAME: __token__
TWINE_PASSWORD: ${{ secrets.PYPI_API_KEY }}
run: |
# Setup Environment for Reproducible Builds
export PYTHONHASHSEED=42
export SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct)
# Build and Upload PyPi Package
rm -rf dist
python -m build
twine check dist/*
twine upload --verbose dist/*
- name: Publish Master to PyPI
if: github.ref == 'refs/heads/master'
env:
TWINE_USERNAME: __token__
TWINE_PASSWORD: ${{ secrets.PYPI_API_KEY }}
run: |
# Set Pre-Release Version
sed -E -i "s/version=(.*)',/version=\1a$(date +%s)',/g" setup.py
# Setup Environment for Reproducible Builds
export PYTHONHASHSEED=42
export SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct)
# Build and Upload PyPi Package
rm -rf dist
python -m build
twine check dist/*
twine upload --verbose dist/*
- name: Publish PR to Test PyPI
if: github.event_name == 'pull_request'
env:
TWINE_USERNAME: __token__
TWINE_PASSWORD: ${{ secrets.TEST_PYPI_API_KEY }}
PULL_REQUEST_NUMBER: ${{ github.event.number }}
run: |
# Set Development Release Version
sed -E -i "s/version=(.*)',/version=\1.dev$PULL_REQUEST_NUMBER$(date +%s)',/g" setup.py
# Setup Environment for Reproducible Builds
export PYTHONHASHSEED=42
export SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct)
# Build and Upload PyPi Package
rm -rf dist
python -m build
twine check dist/*
twine upload -r testpypi --verbose dist/*

64
.github/workflows/pypi.yml vendored Normal file
View File

@@ -0,0 +1,64 @@
name: pypi
on:
push:
tags:
- "*"
branches:
- 'master'
paths:
- src/khoj/**
- pyproject.toml
- .github/workflows/pypi.yml
pull_request:
branches:
- 'master'
paths:
- src/khoj/**
- pyproject.toml
- .github/workflows/pypi.yml
jobs:
publish:
name: Publish Python Package to PyPI
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
with:
fetch-depth: 0
- name: Set up Python 3.10
uses: actions/setup-python@v4
with:
python-version: '3.10'
- name: ⬇️ Install Application
run: python -m pip install --upgrade pip && pip install --upgrade .
- name: ⚙️ Build Python Package
run: |
# Setup Environment for Reproducible Builds
export PYTHONHASHSEED=42
export SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct)
rm -rf dist
# Build PyPi Package
pipx run build
- name: 🌡️ Validate Python Package
run: |
# Validate PyPi Package
pipx run check-wheel-contents dist/*.whl
pipx run twine check dist/*
- name: ⏫ Upload Python Package Artifacts
uses: actions/upload-artifact@v3
with:
name: khoj-assistant
path: dist/*.whl
- name: 📦 Publish Python Package to PyPI
if: startsWith(github.ref, 'refs/tags') || github.ref == 'refs/heads/master'
uses: pypa/gh-action-pypi-publish@v1.6.4
with:
password: ${{ secrets.PYPI_API_KEY }}

75
.github/workflows/release.yml vendored
View File

@@ -12,7 +12,61 @@ on:
type: string
jobs:
publish:
publish_obsidian_plugin:
name: 💎 Publish Obsidian Plugin
runs-on: ubuntu-latest
permissions:
contents: write
defaults:
run:
shell: bash
working-directory: src/interface/obsidian
steps:
- uses: actions/checkout@v3
- name: Install Node
uses: actions/setup-node@v3
with:
node-version: "lts/*"
- name: ⚙️ Build Obsidian Plugin
run: |
yarn
yarn run build --if-present
- name: ⏫ Upload Obsidian Plugin main.js
uses: actions/upload-artifact@v3
with:
if-no-files-found: error
name: main.js
path: src/interface/obsidian/main.js
- name: ⏫ Upload Obsidian Plugin manifest.json
uses: actions/upload-artifact@v3
with:
if-no-files-found: error
name: manifest.json
path: src/interface/obsidian/manifest.json
- name: ⏫ Upload Obsidian Plugin styles.css
uses: actions/upload-artifact@v3
with:
if-no-files-found: error
name: styles.css
path: src/interface/obsidian/styles.css
- name: 🌈 Create Release
uses: softprops/action-gh-release@v1
if: startsWith(github.ref, 'refs/tags/')
with:
generate_release_notes: true
files: |
src/interface/obsidian/main.js
src/interface/obsidian/manifest.json
src/interface/obsidian/styles.css
publish_desktop_apps:
name: 🖥️ Publish Desktop Apps
strategy:
matrix:
include:
@@ -23,6 +77,8 @@ jobs:
- os: windows-latest
extension: exe
runs-on: ${{ matrix.os }}
permissions:
contents: write
steps:
- uses: actions/checkout@v3
@@ -31,7 +87,7 @@ jobs:
with:
python-version: '3.9'
- name: Install Dependencies
- name: ⏬️ Install Dependencies
shell: bash
run: |
if [ "$RUNNER_OS" == "Linux" ]; then
@@ -40,11 +96,11 @@ jobs:
python -m pip install --upgrade pip
pip install pyinstaller
- name: Install Khoj App
- name: ⬇️ Install Khoj App
run: |
pip install --upgrade .
- name: Package Khoj App
- name: 📦 Package Khoj App
shell: bash
run: |
# Setup Environment for Reproducible Builds
@@ -56,7 +112,7 @@ jobs:
mv dist/Khoj.exe dist/khoj_"$GITHUB_REF_NAME"_amd64.exe
fi
- name: Create Mac App DMG
- name: 💻 Create Mac App DMG
if: matrix.os == 'macos-latest'
run: |
# Install Mac DMG Creator
@@ -66,7 +122,7 @@ jobs:
# Create disk image with the app
create-dmg \
--volname "Khoj" \
--volicon "src/interface/web/assets/icons/favicon.icns" \
--volicon "src/khoj/interface/web/assets/icons/favicon.icns" \
--window-pos 200 120 \
--window-size 600 300 \
--icon-size 100 \
@@ -80,7 +136,7 @@ jobs:
if: matrix.os == 'ubuntu-latest'
with:
ruby-version: '3.0'
- name: Create Debian Package
- name: 🐧 Create Debian Package
if: matrix.os == 'ubuntu-latest'
shell: bash
env:
@@ -92,7 +148,7 @@ jobs:
# Copy app files into expected output directory structure
mkdir -p package/opt package/usr/share/applications package/usr/share/icons/hicolor/128x128/apps
cp -r dist/Khoj package/opt/Khoj
cp src/interface/web/assets/icons/favicon-128x128.png package/usr/share/icons/hicolor/128x128/apps/Khoj.png
cp src/khoj/interface/web/assets/icons/favicon-128x128.png package/usr/share/icons/hicolor/128x128/apps/Khoj.png
cp Khoj.desktop package/usr/share/applications
# Fix permissions to be usable by non-root users
@@ -110,8 +166,9 @@ jobs:
name: khoj_${{github.ref_name}}_amd64.${{matrix.extension}}
path: dist/khoj_${{github.ref_name}}_amd64.${{matrix.extension}}
- name: Release
- name: 🌈 Release
uses: softprops/action-gh-release@v1
if: startsWith(github.ref, 'refs/tags/')
with:
generate_release_notes: true
files: dist/khoj_${{github.ref_name}}_amd64.${{matrix.extension}}

38
.github/workflows/test.yml vendored
View File

@@ -5,43 +5,53 @@ on:
branches:
- 'master'
paths:
- src/**
- src/khoj/**
- tests/**
- config/**
- setup.py
- pyproject.toml
- .pre-commit-config.yml
- .github/workflows/test.yml
push:
branches:
- 'master'
paths:
- src/**
- src/khoj/**
- tests/**
- config/**
- setup.py
- pyproject.toml
- .pre-commit-config.yml
- .github/workflows/test.yml
jobs:
test:
name: Run Tests
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
python_version:
- '3.8'
- '3.9'
- '3.10'
- '3.11'
steps:
- uses: actions/checkout@v3
- name: Set up Python 3.10
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: '3.10'
python-version: ${{ matrix.python_version }}
- name: Install Dependencies
- name: ⏬️ Install Dependencies
run: |
sudo apt update && sudo apt install -y libegl1
python -m pip install --upgrade pip
pip install pytest
- name: Install Application
run: |
pip install --upgrade .
- name: ⬇️ Install Application
run: pip install --upgrade .[dev]
- name: Test Application
run: |
pytest
- name: 🌡️ Validate Application
run: pre-commit run --hook-stage manual --all
- name: 🧪 Test Application
run: pytest

52
.github/workflows/test_khoj_el.yml vendored Normal file
View File

@@ -0,0 +1,52 @@
name: test khoj.el
on:
push:
branches:
- 'master'
paths:
- src/interface/emacs/*.el
- src/interface/emacs/tests/*.el
- .github/workflows/test_khoj_el.yml
pull_request:
branches:
- 'master'
paths:
- src/interface/emacs/*.el
- src/interface/emacs/tests/*.el
- .github/workflows/test_khoj_el.yml
jobs:
test:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
emacs_version:
- 27.1
- 27.2
- 28.1
- 28.2
- snapshot
steps:
- uses: purcell/setup-emacs@master
with:
version: ${{ matrix.emacs_version }}
- uses: actions/checkout@v3
- name: 🧪 Test Khoj.el
run: |
# Run ERT tests on khoj.el
emacs -batch \
--eval "(progn \
(require 'package) \
(push '(\"melpa\" . \"https://melpa.org/packages/\") package-archives) \
(package-initialize) \
(unless package-archive-contents (package-refresh-contents)) \
(unless (package-installed-p 'transient) (package-install 'transient)) \
(unless (package-installed-p 'dash) (package-install 'dash)) \
(unless (package-installed-p 'org) (package-install 'org)) \
)" \
-l ert \
-l ./src/interface/emacs/khoj.el \
-l ./src/interface/emacs/tests/khoj-tests.el \
-f ert-run-tests-batch-and-exit

7
.gitignore vendored
View File

@@ -1,7 +1,6 @@
# Khoj artifacts
*.gz
*.pt
src/.data
tests/data/models
tests/data/embeddings
@@ -11,12 +10,14 @@ __pycache__
.emacs.desktop*
*.py[cod]
.vscode
.env
.venv/*
# Build artifacts
/src/interface/web/images
/src/khoj/interface/web/images
/build/
/dist/
/khoj_assistant.egg-info/
khoj_assistant.egg-info
/config/khoj*.yml
.pytest_cache
khoj.log

13
.mypy.ini
View File

@@ -1,13 +0,0 @@
[mypy]
strict_optional = False
ignore_missing_imports = True
install_types = True
non_interactive = True
show_error_codes = True
exclude = (?x)(
src/interface/desktop/main_window.py
| src/interface/desktop/file_browser.py
| src/interface/desktop/system_tray.py
| build/*
| tests/*
)

25
.pre-commit-config.yaml Normal file
View File

@@ -0,0 +1,25 @@
repos:
- repo: https://github.com/psf/black
rev: 23.1.0
hooks:
- id: black
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.4.0
hooks:
- id: end-of-file-fixer
- id: trailing-whitespace
# Exclude elisp files to not clear page breaks
exclude: \.el$
- id: check-json
- id: check-toml
- id: check-yaml
- repo: https://github.com/pre-commit/mirrors-mypy
rev: v1.0.0
hooks:
- id: mypy
stages: [push, manual]
pass_filenames: false
args:
- --config-file=pyproject.toml

14
Dockerfile
View File

@@ -1,18 +1,14 @@
# syntax=docker/dockerfile:1
FROM python:3.10-slim-bullseye
LABEL org.opencontainers.image.source https://github.com/debanjum/khoj
FROM ubuntu:kinetic
LABEL org.opencontainers.image.source https://github.com/khoj-ai/khoj
# Install System Dependencies
RUN apt-get update -y && \
apt-get -y install python3-pyqt5
# Copy Application to Container
COPY . /app
WORKDIR /app
RUN apt update -y && \
apt -y install python3-pip python3-pyqt6 git
# Install Python Dependencies
RUN pip install --upgrade pip && \
pip install --upgrade .
pip install git+https://github.com/khoj-ai/khoj.git
# Run the Application
# There are more arguments required for the application to run,

4
Khoj.desktop
View File

@@ -1,7 +1,7 @@
[Desktop Entry]
Type=Application
Name=Khoj
Comment=A natural language search engine for your personal notes, transactions and images.
Comment=An AI personal assistant for your Digital Brain
Path=/opt
Exec=/opt/Khoj
Icon=Khoj
Icon=Khoj

20
Khoj.spec
View File

@@ -5,9 +5,12 @@ from PyInstaller.utils.hooks import copy_metadata
import sysconfig
datas = [
('src/interface/web', 'src/interface/web'),
(f'{sysconfig.get_paths()["purelib"]}/transformers', 'transformers')
('src/khoj/interface/web', 'khoj/interface/web'),
(f'{sysconfig.get_paths()["purelib"]}/transformers', 'transformers'),
(f'{sysconfig.get_paths()["purelib"]}/langchain', 'langchain'),
(f'{sysconfig.get_paths()["purelib"]}/PIL', 'PIL')
]
datas += copy_metadata('torch')
datas += copy_metadata('tqdm')
datas += copy_metadata('regex')
datas += copy_metadata('requests')
@@ -15,15 +18,16 @@ datas += copy_metadata('packaging')
datas += copy_metadata('filelock')
datas += copy_metadata('numpy')
datas += copy_metadata('tokenizers')
datas += copy_metadata('pillow')
block_cipher = None
a = Analysis(
['src/main.py'],
['src/khoj/main.py'],
pathex=[],
binaries=[],
datas=datas,
hiddenimports=['huggingface_hub.repository'],
hiddenimports=['huggingface_hub.repository', 'PIL', 'PIL._tkinter_finder', 'tiktoken_ext', 'tiktoken_ext.openai_public'],
hookspath=[],
hooksconfig={},
runtime_hooks=[],
@@ -50,7 +54,7 @@ pyz = PYZ(a.pure, a.zipped_data, cipher=block_cipher)
if system() != 'Darwin':
# Add Splash screen to show on app launch
splash = Splash(
'src/interface/web/assets/icons/favicon-144x144.png',
'src/khoj/interface/web/assets/icons/favicon-128x128.png',
binaries=a.binaries,
datas=a.datas,
text_pos=(10, 160),
@@ -82,7 +86,7 @@ if system() != 'Darwin':
target_arch='x86_64',
codesign_identity=None,
entitlements_file=None,
icon='src/interface/web/assets/icons/favicon-144x144.ico',
icon='src/khoj/interface/web/assets/icons/favicon-128x128.ico',
)
else:
exe = EXE(
@@ -105,11 +109,11 @@ else:
target_arch='x86_64',
codesign_identity=None,
entitlements_file=None,
icon='src/interface/web/assets/icons/favicon.icns',
icon='src/khoj/interface/web/assets/icons/favicon.icns',
)
app = BUNDLE(
exe,
name='Khoj.app',
icon='src/interface/web/assets/icons/favicon.icns',
icon='src/khoj/interface/web/assets/icons/favicon.icns',
bundle_identifier=None,
)

1
LICENSE
View File

@@ -619,4 +619,3 @@ Program, unless a warranty or assumption of liability accompanies a
copy of the Program in return for a fee.
END OF TERMS AND CONDITIONS

5
MANIFEST.in
View File

@@ -1,5 +0,0 @@
include Readme.md
graft src/interface/*
prune src/interface/web/images*
prune docs*
global-exclude .DS_Store *.py[cod]

525
README.md Normal file
View File

@@ -0,0 +1,525 @@
<h1><img src="src/khoj/interface/web/assets/icons/khoj-logo-sideways.svg" width="330" alt="Khoj Logo"></h1>
[![test](https://github.com/khoj-ai/khoj/actions/workflows/test.yml/badge.svg)](https://github.com/khoj-ai/khoj/actions/workflows/test.yml)
[![dockerize](https://github.com/khoj-ai/khoj/actions/workflows/dockerize.yml/badge.svg)](https://github.com/khoj-ai/khoj/pkgs/container/khoj)
[![pypi](https://github.com/khoj-ai/khoj/actions/workflows/pypi.yml/badge.svg)](https://pypi.org/project/khoj-assistant/)
*An AI personal assistant for your digital brain*
**Supported Plugins**
[![Khoj on Obsidian](https://img.shields.io/badge/Obsidian-%23483699.svg?style=for-the-badge&logo=obsidian&logoColor=white)](https://github.com/khoj-ai/khoj/tree/master/src/interface/obsidian#readme)
[![Khoj on Emacs](https://img.shields.io/badge/Emacs-%237F5AB6.svg?&style=for-the-badge&logo=gnu-emacs&logoColor=white)](https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#readme)
## Table of Contents
- [Features](#Features)
- [Demos](#Demos)
- [Khoj in Obsidian](#khoj-in-obsidian)
- [Khoj in Emacs, Browser](#khoj-in-emacs-browser)
- [Interfaces](#Interfaces)
- [Architecture](#Architecture)
- [Setup](#Setup)
- [Install](#1-Install)
- [Run](#2-Run)
- [Configure](#3-Configure)
- [Install Plugins](#4-install-interface-plugins)
- [Use](#Use)
- [Khoj Search](#Khoj-search)
- [Khoj Chat](#Khoj-chat)
- [Upgrade](#Upgrade)
- [Khoj Server](#upgrade-khoj-server)
- [Khoj.el](#upgrade-khoj-on-emacs)
- [Khoj Obsidian](#upgrade-khoj-on-obsidian)
- [Uninstall](#uninstall)
- [Troubleshoot](#Troubleshoot)
- [Advanced Usage](#advanced-usage)
- [Access Khoj on Mobile](#access-khoj-on-mobile)
- [Use OpenAI Models for Search](#use-openai-models-for-search)
- [Search across Different Languages](#search-across-different-languages)
- [Boostrap Khoj Search for Offline Usage Later](#bootstrap-khoj-search-for-offline-usage-later)
- [Miscellaneous](#Miscellaneous)
- [Setup OpenAI API key in Khoj](#set-your-openai-api-key-in-khoj)
- [GPT API](#gpt-api)
- [Performance](#Performance)
- [Query Performance](#Query-performance)
- [Indexing Performance](#Indexing-performance)
- [Miscellaneous](#Miscellaneous-1)
- [Development](#Development)
- [Visualize Codebase](#visualize-codebase)
- [Setup](#Setup)
- [Using Pip](#Using-Pip)
- [Using Docker](#Using-Docker)
- [Using Conda](#Using-Conda)
- [Validate](#Validate)
- [Credits](#Credits)
## Features
- **Search**
- **Local**: Your personal data stays local. All search and indexing is done on your machine. *Unlike chat which requires access to GPT.*
- **Incremental**: Incremental search for a fast, search-as-you-type experience
- **Chat**
- **Faster answers**: Find answers faster, smoother than search. No need to manually scan through your notes to find answers.
- **Iterative discovery**: Iteratively explore and (re-)discover your notes
- **Assisted creativity**: Smoothly weave across answers retrieval and content generation
- **General**
- **Natural**: Advanced natural language understanding using Transformer based ML Models
- **Pluggable**: Modular architecture makes it easy to plug in new data sources, frontends and ML models
- **Multiple Sources**: Index your Org-mode and Markdown notes, PDF files, Github repositories, and Photos
- **Multiple Interfaces**: Interact from your [Web Browser](./src/khoj/interface/web/index.html), [Emacs](./src/interface/emacs/khoj.el) or [Obsidian](./src/interface/obsidian/)
## Demos
### Khoj in Obsidian
https://github.com/khoj-ai/khoj/assets/6413477/3e33d8ea-25bb-46c8-a3bf-c92f78d0f56b
<details><summary>Description</summary>
1. Install Khoj via `pip` and start Khoj backend in a terminal (Run `khoj`)
```
python -m pip install khoj-assistant
khoj
```
2. Install Khoj plugin via Community Plugins settings pane on Obsidian app
- Check the new Khoj plugin settings
- Let Khoj backend index the markdown, pdf, Github markdown files in the current Vault
- Open Khoj plugin on Obsidian via Search button on Left Pane
- Search \"*Announce plugin to folks*\" in the [Obsidian Plugin docs](https://marcus.se.net/obsidian-plugin-docs/)
- Jump to the [search result](https://marcus.se.net/obsidian-plugin-docs/publishing/submit-your-plugin)
</details>
### Khoj in Emacs, Browser
https://user-images.githubusercontent.com/6413477/184735169-92c78bf1-d827-4663-9087-a1ea194b8f4b.mp4
<details><summary>Description</summary>
- Install Khoj via pip
- Start Khoj app
- Add this readme and [khoj.el readme](https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs) as org-mode for Khoj to index
- Search \"*Setup editor*\" on the Web and Emacs. Re-rank the results for better accuracy
- Top result is what we are looking for, the [section to Install Khoj.el on Emacs](https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#2-Install-Khojel)
</details>
<details><summary>Analysis</summary>
- The results do not have any words used in the query
- *Based on the top result it seems the re-ranking model understands that Emacs is an editor?*
- The results incrementally update as the query is entered
- The results are re-ranked, for better accuracy, once user hits enter
</details>
### Interfaces
![](https://github.com/khoj-ai/khoj/blob/master/docs/interfaces.png?)
## Architecture
![](https://github.com/khoj-ai/khoj/blob/master/docs/khoj_architecture.png?)
## Setup
These are the general setup instructions for Khoj.
- Make sure [python](https://realpython.com/installing-python/) and [pip](https://pip.pypa.io/en/stable/installation/) are installed on your machine
- Check the [Khoj.el Readme](https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Setup) to setup Khoj with Emacs<br />
Its simpler as it can skip the server *install*, *run* and *configure* step below.
- Check the [Khoj Obsidian Readme](https://github.com/khoj-ai/khoj/tree/master/src/interface/obsidian#Setup) to setup Khoj with Obsidian<br />
Its simpler as it can skip the *configure* step below.
### 1. Install
Run the following command in your terminal to install the Khoj backend.
- On Linux/MacOS
```shell
python -m pip install khoj-assistant
```
- On Windows
```shell
py -m pip install khoj-assistant
```
### 2. Run
Run the following commmand from your terminal to start the Khoj backend and open Khoj in your browser.
```shell
khoj --gui
```
Note: To start Khoj automatically in the background use [Task scheduler](https://www.windowscentral.com/how-create-automated-task-using-task-scheduler-windows-10) on Windows or [Cron](https://en.wikipedia.org/wiki/Cron) on Mac, Linux (e.g with `@reboot khoj`)
### 3. Configure
1. Set `File`, `Folder` and hit `Save` in each Plugins you want to enable for Search on the Khoj config page
2. Add your OpenAI API key to Chat Feature settings if you want to use Chat
3. Click `Configure` and wait. The app will download ML models and index the content for search and (optionally) chat
### 4. Install Interface Plugins
Khoj exposes a web interface to search, chat and configure by default.<br />
The optional steps below allow using Khoj from within an existing application like Obsidian or Emacs.
- **Khoj Obsidian**:<br />
[Install](https://github.com/khoj-ai/khoj/tree/master/src/interface/obsidian#2-Setup-Plugin) the Khoj Obsidian plugin
- **Khoj Emacs**:<br />
[Install](https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#2-Install-Khojel) khoj.el
## Use
### Khoj Search
- **Khoj via Obsidian**
- Click the *Khoj search* icon 🔎 on the [Ribbon](https://help.obsidian.md/User+interface/Workspace/Ribbon) or Search for *Khoj: Search* in the [Command Palette](https://help.obsidian.md/Plugins/Command+palette)
- **Khoj via Emacs**
- Run `M-x khoj <user-query>`
- **Khoj via Web**
- Open <http://localhost:8000/> directly
- **Khoj via API**
- See the Khoj FastAPI [Swagger Docs](http://localhost:8000/docs), [ReDocs](http://localhost:8000/redocs)
<details><summary>Query Filters</summary>
Use structured query syntax to filter the natural language search results
- **Word Filter**: Get entries that include/exclude a specified term
- Entries that contain term_to_include: `+"term_to_include"`
- Entries that contain term_to_exclude: `-"term_to_exclude"`
- **Date Filter**: Get entries containing dates in YYYY-MM-DD format from specified date (range)
- Entries from April 1st 1984: `dt:"1984-04-01"`
- Entries after March 31st 1984: `dt>="1984-04-01"`
- Entries before April 2nd 1984 : `dt<="1984-04-01"`
- **File Filter**: Get entries from a specified file
- Entries from incoming.org file: `file:"incoming.org"`
- Combined Example
- `what is the meaning of life? file:"1984.org" dt>="1984-01-01" dt<="1985-01-01" -"big" -"brother"`
- Adds all filters to the natural language query. It should return entries
- from the file *1984.org*
- containing dates from the year *1984*
- excluding words *"big"* and *"brother"*
- that best match the natural language query *"what is the meaning of life?"*
</details>
### Khoj Chat
#### Overview
- Creates a personal assistant for you to inquire and engage with your notes
- Uses [ChatGPT](https://openai.com/blog/chatgpt) and [Khoj search](#khoj-search)
- Supports multi-turn conversations with the relevant notes for context
- Shows reference notes used to generate a response
- **Note**: *Your query and top notes from khoj search will be sent to OpenAI for processing*
#### Setup
- [Setup your OpenAI API key in Khoj](#set-your-openai-api-key-in-khoj)
#### Use
1. Open [/chat](http://localhost:8000/chat)[^2]
2. Type your queries and see response by Khoj from your notes
#### Demo
![](https://github.com/khoj-ai/khoj/blob/master/docs/khoj_chat_web_interface.png?)
### Details
1. Your query is used to retrieve the most relevant notes, if any, using Khoj search
2. These notes, the last few messages and associated metadata is passed to ChatGPT along with your query for a response
## Upgrade
### Upgrade Khoj Server
```shell
pip install --upgrade khoj-assistant
```
*Note: To upgrade to the latest pre-release version of the khoj server run below command*
```shell
# Maps to the latest commit on the master branch
pip install --upgrade --pre khoj-assistant
```
### Upgrade Khoj on Emacs
- Use your Emacs Package Manager to Upgrade
- See [khoj.el readme](https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Upgrade) for details
### Upgrade Khoj on Obsidian
- Upgrade via the Community plugins tab on the settings pane in the Obsidian app
- See the [khoj plugin readme](https://github.com/khoj-ai/khoj/tree/master/src/interface/obsidian#2-Setup-Plugin) for details
## Uninstall
1. (Optional) Hit `Ctrl-C` in the terminal running the khoj server to stop it
2. Delete the khoj directory in your home folder (i.e `~/.khoj` on Linux, Mac or `C:\Users\<your-username>\.khoj` on Windows)
3. Uninstall the khoj server with `pip uninstall khoj-assistant`
4. (Optional) Uninstall khoj.el or the khoj obsidian plugin in the standard way on Emacs, Obsidian
## Troubleshoot
#### Install fails while building Tokenizer dependency
- **Details**: `pip install khoj-assistant` fails while building the `tokenizers` dependency. Complains about Rust.
- **Fix**: Install Rust to build the tokenizers package. For example on Mac run:
```shell
brew install rustup
rustup-init
source ~/.cargo/env
```
- **Refer**: [Issue with Fix](https://github.com/khoj-ai/khoj/issues/82#issuecomment-1241890946) for more details
#### Search starts giving wonky results
- **Fix**: Open [/api/update?force=true](http://localhost:8000/api/update?force=true)[^2] in browser to regenerate index from scratch
- **Note**: *This is a fix for when you percieve the search results have degraded. Not if you think they've always given wonky results*
#### Khoj in Docker errors out with \"Killed\" in error message
- **Fix**: Increase RAM available to Docker Containers in Docker Settings
- **Refer**: [StackOverflow Solution](https://stackoverflow.com/a/50770267), [Configure Resources on Docker for Mac](https://docs.docker.com/desktop/mac/#resources)
#### Khoj errors out complaining about Tensors mismatch or null
- **Mitigation**: Disable `image` search using the desktop GUI
## Advanced Usage
### Access Khoj on Mobile
1. [Setup Khoj](#Setup) on your personal server. This can be any always-on machine, i.e an old computer, RaspberryPi(?) etc
2. [Install](https://tailscale.com/kb/installation/) [Tailscale](tailscale.com/) on your personal server and phone
3. Open the Khoj web interface of the server from your phone browser.<br /> It should be `http://tailscale-ip-of-server:8000` or `http://name-of-server:8000` if you've setup [MagicDNS](https://tailscale.com/kb/1081/magicdns/)
4. Click the [Add to Homescreen](https://developer.mozilla.org/en-US/docs/Web/Progressive_web_apps/Add_to_home_screen) button
5. Enjoy exploring your notes, documents and images from your phone!
![](https://github.com/khoj-ai/khoj/blob/master/docs/khoj_pwa_android.png?)
### Use OpenAI Models for Search
#### Setup
1. Set `encoder-type`, `encoder` and `model-directory` under `asymmetric` and/or `symmetric` `search-type` in your `khoj.yml`[^1]:
```diff
asymmetric:
- encoder: "sentence-transformers/multi-qa-MiniLM-L6-cos-v1"
+ encoder: text-embedding-ada-002
+ encoder-type: khoj.utils.models.OpenAI
cross-encoder: "cross-encoder/ms-marco-MiniLM-L-6-v2"
- encoder-type: sentence_transformers.SentenceTransformer
- model_directory: "~/.khoj/search/asymmetric/"
+ model-directory: null
```
2. [Setup your OpenAI API key in Khoj](#set-your-openai-api-key-in-khoj)
3. Restart Khoj server to generate embeddings. It will take longer than with offline models.
#### Warnings
This configuration *uses an online model*
- It will **send all notes to OpenAI** to generate embeddings
- **All queries will be sent to OpenAI** when you search with Khoj
- You will be **charged by OpenAI** based on the total tokens processed
- It *requires an active internet connection* to search and index
### Search across Different Languages
To search for notes in multiple, different languages, you can use a [multi-lingual model](https://www.sbert.net/docs/pretrained_models.html#multi-lingual-models).<br />
For example, the [paraphrase-multilingual-MiniLM-L12-v2](https://huggingface.co/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2) supports [50+ languages](https://www.sbert.net/docs/pretrained_models.html#:~:text=we%20used%20the%20following%2050%2B%20languages), has good search quality and speed. To use it:
1. Manually update `search-type > asymmetric > encoder` to `paraphrase-multilingual-MiniLM-L12-v2` in your `~/.khoj/khoj.yml` file for now. See diff of `khoj.yml` below for illustration:
```diff
asymmetric:
- encoder: "sentence-transformers/multi-qa-MiniLM-L6-cos-v1"
+ encoder: "paraphrase-multilingual-MiniLM-L12-v2"
cross-encoder: "cross-encoder/ms-marco-MiniLM-L-6-v2"
model_directory: "~/.khoj/search/asymmetric/"
```
2. Regenerate your content index. For example, by opening [\<khoj-url\>/api/update?t=force](http://localhost:8000/api/update?t=force)
### Bootstrap Khoj Search for Offline Usage later
You can bootstrap Khoj pre-emptively to run on machines that do not have internet access. An example use-case would be to run Khoj on an air-gapped machine.
Note: *Only search can currently run in fully offline mode, not chat.*
- With Internet
1. Manually download the [asymmetric text](https://huggingface.co/sentence-transformers/multi-qa-MiniLM-L6-cos-v1), [symmetric text](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2)and [image search](https://huggingface.co/sentence-transformers/clip-ViT-B-32) models from HuggingFace
2. Pip install khoj (and dependencies) in an associated virtualenv. E.g `python -m venv .venv && source .venv/bin/activate && pip install khoj-assistant`
- Without Internet
1. Copy each of the search models into their respective folders, `asymmetric`, `symmetric` and `image` under the `~/.khoj/search/` directory on the air-gapped machine
2. Copy the khoj virtual environment directory onto the air-gapped machine, activate the environment and start and khoj as normal. E.g `source .venv/bin/activate && khoj`
## Miscellaneous
### Set your OpenAI API key in Khoj
If you want, Khoj can be configured to use OpenAI for search and chat.<br />
Add your OpenAI API to Khoj by using either of the two options below:
- Open your [Khoj settings](http://localhost:8000/config/processor/conversation), add your OpenAI API key, and click *Save*. Then go to your [Khoj settings](http://localhost:8000/config) and click `Configure`. This will refresh Khoj with your OpenAI API key.
- Set `openai-api-key` field under `processor.conversation` section in your `khoj.yml`[^1] to your [OpenAI API key](https://beta.openai.com/account/api-keys) and restart khoj:
```diff
processor:
conversation:
- openai-api-key: # "YOUR_OPENAI_API_KEY"
+ openai-api-key: sk-aaaaaaaaaaaaaaaaaaaaaaaahhhhhhhhhhhhhhhhhhhhhhhh
model: "text-davinci-003"
conversation-logfile: "~/.khoj/processor/conversation/conversation_logs.json"
```
**Warning**: *This will enable Khoj to send your query and note(s) to OpenAI for processing*
### GPT API
- The [chat](http://localhost:8000/api/chat), [answer](http://localhost:8000/api/beta/answer) and [search](http://localhost:8000/api/beta/search) API endpoints use [OpenAI API](https://openai.com/api/)
- They are disabled by default
- To use them:
1. [Setup your OpenAI API key in Khoj](#set-your-openai-api-key-in-khoj)
2. Interact with them from the [Khoj Swagger docs](http://locahost:8000/docs)[^2]
### Index Github Repository for Search, Chat
The Khoj Github plugin can index issues, commit messages and markdown, org-mode and PDF files from any repositories you have access to. This allows you to chat or search with these repositories. Get answers, resolve issues or just explore a repo with the help of your AI personal assistant.
See the [Khoj FAQ](https://faq.khoj.dev) for a demo of Khoj search and chat. It makes the Khoj github repo available for exploring.
Note: *Khoj will ignore code files in the repository for now as the default AI model used works best with natural language text, not code.*
#### Setup Khoj Github plugin
1. Get a [pat token](https://docs.github.com/en/github/authenticating-to-github/keeping-your-account-and-data-secure/creating-a-personal-access-token) with `repo` and `read:org` scopes in the classic flow.
2. Configure Khoj settings to include the `owner` and `repo_name`. The `owner` will be the organization name if the repo is in an organization. The `repo_name` will be the name of the repository. Optionally, you can also supply a branch name. If no branch name is supplied, the `master` branch will be used.
## Performance
### Query performance
- Semantic search using the bi-encoder is fairly fast at \<50 ms
- Reranking using the cross-encoder is slower at \<2s on 15 results. Tweak `top_k` to tradeoff speed for accuracy of results
- Filters in query (e.g by file, word or date) usually add \<20ms to query latency
### Indexing performance
- Indexing is more strongly impacted by the size of the source data
- Indexing 100K+ line corpus of notes takes about 10 minutes
- Indexing 4000+ images takes about 15 minutes and more than 8Gb of RAM
- Note: *It should only take this long on the first run* as the index is incrementally updated
### Miscellaneous
- Testing done on a Mac M1 and a \>100K line corpus of notes
- Search, indexing on a GPU has not been tested yet
## Development
### Visualize Codebase
*[Interactive Visualization](https://mango-dune-07a8b7110.1.azurestaticapps.net/?repo=debanjum%2Fkhoj)*
![](https://github.com/khoj-ai/khoj/blob/master/docs/khoj_codebase_visualization_0.2.1.png?)
### Setup
#### Using Pip
##### 1. Install
```shell
# Get Khoj Code
git clone https://github.com/khoj-ai/khoj && cd khoj
# Create, Activate Virtual Environment
python3 -m venv .venv && source .venv/bin/activate
# Install Khoj for Development
pip install -e .[dev]
```
##### 2. Run
1. Start Khoj
```shell
khoj -vv
```
2. Configure Khoj
- **Via the Settings UI**: Add files, directories to index the [Khoj settings](http://localhost:8000/config) UI once Khoj has started up. Once you've saved all your settings, click `Configure`.
- **Manually**:
- Copy the `config/khoj_sample.yml` to `~/.khoj/khoj.yml`
- Set `input-files` or `input-filter` in each relevant `content-type` section of `~/.khoj/khoj.yml`
- Set `input-directories` field in `image` `content-type` section
- Delete `content-type` and `processor` sub-section(s) irrelevant for your use-case
- Restart khoj
Note: Wait after configuration for khoj to Load ML model, generate embeddings and expose API to query notes, images, documents etc specified in config YAML
#### Using Docker
##### 1. Clone
```shell
git clone https://github.com/khoj-ai/khoj && cd khoj
```
##### 2. Configure
- **Required**: Update [docker-compose.yml](./docker-compose.yml) to mount your images, (org-mode or markdown) notes, PDFs and Github repositories
- **Optional**: Edit application configuration in [khoj_docker.yml](./config/khoj_docker.yml)
##### 3. Run
```shell
docker-compose up -d
```
*Note: The first run will take time. Let it run, it\'s mostly not hung, just generating embeddings*
##### 4. Upgrade
```shell
docker-compose build --pull
```
#### Using Conda
##### 1. Install Dependencies
- [Install Conda](https://docs.conda.io/projects/conda/en/latest/user-guide/install/index.html)
##### 2. Install Khoj
```shell
git clone https://github.com/khoj-ai/khoj && cd khoj
conda env create -f config/environment.yml
conda activate khoj
python3 -m pip install pyqt6 # As conda does not support pyqt6 yet
```
##### 3. Configure
- Copy the `config/khoj_sample.yml` to `~/.khoj/khoj.yml`
- Set `input-files` or `input-filter` in each relevant `content-type` section of `~/.khoj/khoj.yml`
- Set `input-directories` field in `image` `content-type` section
- Delete `content-type`, `processor` sub-sections irrelevant for your use-case
##### 4. Run
```shell
python3 -m src.khoj.main -vv
```
Load ML model, generate embeddings and expose API to query notes, images, documents etc specified in config YAML
##### 5. Upgrade
```shell
cd khoj
git pull origin master
conda deactivate khoj
conda env update -f config/environment.yml
conda activate khoj
```
### Validate
#### Before Make Changes
1. Install Git Hooks for Validation
```shell
pre-commit install -t pre-push -t pre-commit
```
- This ensures standard code formatting fixes and other checks run automatically on every commit and push
- Note 1: If [pre-commit](https://pre-commit.com/#intro) didn't already get installed, [install it](https://pre-commit.com/#install) via `pip install pre-commit`
- Note 2: To run the pre-commit changes manually, use `pre-commit run --hook-stage manual --all` before creating PR
#### Before Creating PR
1. Run Tests. If you get an error complaining about a missing `fast_tokenizer_file`, follow the solution [in this Github issue](https://github.com/UKPLab/sentence-transformers/issues/1659).
```shell
pytest
```
2. Run MyPy to check types
```shell
mypy --config-file pyproject.toml
```
#### After Creating PR
- Automated [validation workflows](.github/workflows) run for every PR.
Ensure any issues seen by them our fixed
- Test the python packge created for a PR
1. Download and extract the zipped `.whl` artifact generated from the pypi workflow run for the PR.
2. Install (in your virtualenv) with `pip install /path/to/download*.whl>`
3. Start and use the application to see if it works fine
## Credits
- [Multi-QA MiniLM Model](https://huggingface.co/sentence-transformers/multi-qa-MiniLM-L6-cos-v1), [All MiniLM Model](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) for Text Search. See [SBert Documentation](https://www.sbert.net/examples/applications/retrieve_rerank/README.html)
- [OpenAI CLIP Model](https://github.com/openai/CLIP) for Image Search. See [SBert Documentation](https://www.sbert.net/examples/applications/image-search/README.html)
- Charles Cave for [OrgNode Parser](http://members.optusnet.com.au/~charles57/GTD/orgnode.html)
- [Org.js](https://mooz.github.io/org-js/) to render Org-mode results on the Web interface
- [Markdown-it](https://github.com/markdown-it/markdown-it) to render Markdown results on the Web interface
[^1]: Default Khoj config file @ `~/.khoj/khoj.yml`
[^2]: Default Khoj url @ http://localhost:8000

414
Readme.md
View File

@@ -1,414 +0,0 @@
# Khoj 🦅
[![build](https://github.com/debanjum/khoj/actions/workflows/build.yml/badge.svg)](https://github.com/debanjum/khoj/actions/workflows/build.yml)
[![test](https://github.com/debanjum/khoj/actions/workflows/test.yml/badge.svg)](https://github.com/debanjum/khoj/actions/workflows/test.yml)
[![publish](https://github.com/debanjum/khoj/actions/workflows/publish.yml/badge.svg)](https://github.com/debanjum/khoj/actions/workflows/publish.yml)
*A natural language search engine for your personal notes, transactions and images*
## Table of Contents
- [Features](#Features)
- [Demos](#Demos)
- [Khoj in Obsidian](#khoj-in-obsidian)
- [Khoj in Emacs, Browser](#khoj-in-emacs-browser)
- [Interfaces](#Interfaces)
- [Architecture](#Architecture)
- [Setup](#Setup)
- [Install](#1-Install)
- [Configure](#2-Configure)
- [Run](#3-Run)
- [Use](#Use)
- [Interfaces](#Interfaces-1)
- [Query Filters](#Query-filters)
- [Upgrade](#Upgrade)
- [Khoj Server](#upgrade-khoj-server)
- [Khoj.el](#upgrade-khoj-on-emacs)
- [Khoj Obsidian](#upgrade-khoj-on-obsidian)
- [Troubleshoot](#Troubleshoot)
- [Advanced Usage](#advanced-usage)
- [Access Khoj on Mobile](#access-khoj-on-mobile)
- [Chat with Notes](#chat-with-notes)
- [Use OpenAI Models for Search](#use-openai-models-for-search)
- [Miscellaneous](#Miscellaneous)
- [Setup OpenAI API key in Khoj](#set-your-openai-api-key-in-khoj)
- [Beta API](#beta-api)
- [Performance](#Performance)
- [Query Performance](#Query-performance)
- [Indexing Performance](#Indexing-performance)
- [Miscellaneous](#Miscellaneous-1)
- [Development](#Development)
- [Visualize Codebase](#visualize-codebase)
- [Setup](#Setup)
- [Using Pip](#Using-Pip)
- [Using Docker](#Using-Docker)
- [Using Conda](#Test)
- [Test](#Test)
- [Credits](#Credits)
## Features
- **Natural**: Advanced natural language understanding using Transformer based ML Models
- **Local**: Your personal data stays local. All search, indexing is done on your machine[\*](https://github.com/debanjum/khoj#beta-api)
- **Incremental**: Incremental search for a fast, search-as-you-type experience
- **Pluggable**: Modular architecture makes it easy to plug in new data sources, frontends and ML models
- **Multiple Sources**: Search your Org-mode and Markdown notes, Beancount transactions and Photos
- **Multiple Interfaces**: Search using a [Web Browser](./src/interface/web/index.html), [Emacs](./src/interface/emacs/khoj.el) or the [API](http://localhost:8000/docs)
## Demos
### Khoj in Obsidian
https://user-images.githubusercontent.com/6413477/210486007-36ee3407-e6aa-4185-8a26-b0bfc0a4344f.mp4
<details><summary>Description</summary>
- Install Khoj via `pip` and start Khoj backend in non-gui mode
- Install Khoj plugin via Community Plugins settings pane on Obsidian app
- Check the new Khoj plugin settings
- Let Khoj backend index the markdown files in the current Vault
- Open Khoj plugin on Obsidian via Search button on Left Pane
- Search \"*Announce plugin to folks*\" in the [Obsidian Plugin docs](https://marcus.se.net/obsidian-plugin-docs/)
- Jump to the [search result](https://marcus.se.net/obsidian-plugin-docs/publishing/submit-your-plugin)
</details>
### Khoj in Emacs, Browser
https://user-images.githubusercontent.com/6413477/184735169-92c78bf1-d827-4663-9087-a1ea194b8f4b.mp4
<details><summary>Description</summary>
- Install Khoj via pip
- Start Khoj app
- Add this readme and [khoj.el readme](https://github.com/debanjum/khoj/tree/master/src/interface/emacs) as org-mode for Khoj to index
- Search \"*Setup editor*\" on the Web and Emacs. Re-rank the results for better accuracy
- Top result is what we are looking for, the [section to Install Khoj.el on Emacs](https://github.com/debanjum/khoj/tree/master/src/interface/emacs#2-Install-Khojel)
</details>
<details><summary>Analysis</summary>
- The results do not have any words used in the query
- *Based on the top result it seems the re-ranking model understands that Emacs is an editor?*
- The results incrementally update as the query is entered
- The results are re-ranked, for better accuracy, once user hits enter
</details>
### Interfaces
![](https://github.com/debanjum/khoj/blob/master/docs/interfaces.png?)
## Architecture
![](https://github.com/debanjum/khoj/blob/master/docs/khoj_architecture.png?)
## Setup
These are the general setup instructions for Khoj.
- Check the [Khoj.el Readme](https://github.com/debanjum/khoj/tree/master/src/interface/emacs#Setup) to setup Khoj with Emacs
- Check the [Khoj Obsidian Readme](https://github.com/debanjum/khoj/tree/master/src/interface/obsidian#Setup) to setup Khoj with Obsidian<br />
Its simpler as it can skip the configure step below.
### 1. Install
```shell
pip install khoj-assistant
```
### 2. Start App
```shell
khoj
```
### 3. Configure
1. Enable content types and point to files to search in the First Run Screen that pops up on app start
2. Click `Configure` and wait. The app will download ML models and index the content for search
## Use
### Interfaces
- **Khoj via Obsidian**
- [Install](https://github.com/debanjum/khoj/tree/master/src/interface/obsidian#2-Setup-Plugin) the Khoj Obsidian plugin
- Click the *Khoj search* icon 🔎 on the [Ribbon](https://help.obsidian.md/User+interface/Workspace/Ribbon) or Search for *Khoj: Search* in the [Command Palette](https://help.obsidian.md/Plugins/Command+palette)
- **Khoj via Emacs**
- [Install](https://github.com/debanjum/khoj/tree/master/src/interface/emacs#installation) [khoj.el](./src/interface/emacs/khoj.el)
- Run `M-x khoj <user-query>`
- **Khoj via Web**
- Open <http://localhost:8000/> via desktop interface or directly
- **Khoj via API**
- See the Khoj FastAPI [Swagger Docs](http://localhost:8000/docs), [ReDocs](http://localhost:8000/redocs)
### Query Filters
Use structured query syntax to filter the natural language search results
- **Word Filter**: Get entries that include/exclude a specified term
- Entries that contain term_to_include: `+"term_to_include"`
- Entries that contain term_to_exclude: `-"term_to_exclude"`
- **Date Filter**: Get entries containing dates in YYYY-MM-DD format from specified date (range)
- Entries from April 1st 1984: `dt:"1984-04-01"`
- Entries after March 31st 1984: `dt>="1984-04-01"`
- Entries before April 2nd 1984 : `dt<="1984-04-01"`
- **File Filter**: Get entries from a specified file
- Entries from incoming.org file: `file:"incoming.org"`
- Combined Example
- `what is the meaning of life? file:"1984.org" dt>="1984-01-01" dt<="1985-01-01" -"big" -"brother"`
- Adds all filters to the natural language query. It should return entries
- from the file *1984.org*
- containing dates from the year *1984*
- excluding words *"big"* and *"brother"*
- that best match the natural language query *"what is the meaning of life?"*
## Upgrade
### Upgrade Khoj Server
```shell
pip install --upgrade khoj-assistant
```
### Upgrade Khoj on Emacs
- Use your Emacs Package Manager to Upgrade
- See [khoj.el readme](https://github.com/debanjum/khoj/tree/master/src/interface/emacs#Upgrade) for details
### Upgrade Khoj on Obsidian
- Upgrade via the Community plugins tab on the settings pane in the Obsidian app
- See the [khoj plugin readme](https://github.com/debanjum/khoj/tree/master/src/interface/obsidian#2-Setup-Plugin) for details
## Troubleshoot
#### Install fails while building Tokenizer dependency
- **Details**: `pip install khoj-assistant` fails while building the `tokenizers` dependency. Complains about Rust.
- **Fix**: Install Rust to build the tokenizers package. For example on Mac run:
```shell
brew install rustup
rustup-init
source ~/.cargo/env
```
- **Refer**: [Issue with Fix](https://github.com/debanjum/khoj/issues/82#issuecomment-1241890946) for more details
#### Search starts giving wonky results
- **Fix**: Open [/api/update?force=true](http://localhost:8000/api/update?force=true)[^2] in browser to regenerate index from scratch
- **Note**: *This is a fix for when you percieve the search results have degraded. Not if you think they've always given wonky results*
#### Khoj in Docker errors out with \"Killed\" in error message
- **Fix**: Increase RAM available to Docker Containers in Docker Settings
- **Refer**: [StackOverflow Solution](https://stackoverflow.com/a/50770267), [Configure Resources on Docker for Mac](https://docs.docker.com/desktop/mac/#resources)
#### Khoj errors out complaining about Tensors mismatch or null
- **Mitigation**: Disable `image` search using the desktop GUI
## Advanced Usage
### Access Khoj on Mobile
1. [Setup Khoj](#Setup) on your personal server. This can be any always-on machine, i.e an old computer, RaspberryPi(?) etc
2. [Install](https://tailscale.com/kb/installation/) [Tailscale](tailscale.com/) on your personal server and phone
3. Open the Khoj web interface of the server from your phone browser.<br /> It should be `http://tailscale-ip-of-server:8000` or `http://name-of-server:8000` if you've setup [MagicDNS](https://tailscale.com/kb/1081/magicdns/)
4. Click the [Add to Homescreen](https://developer.mozilla.org/en-US/docs/Web/Progressive_web_apps/Add_to_home_screen) button
5. Enjoy exploring your notes, transactions and images from your phone!
![](https://github.com/debanjum/khoj/blob/master/docs/khoj_pwa_android.png?)
### Chat with Notes
#### Overview
- Provides a chat interface to inquire and engage with your notes
- Chat Types:
- **Summarize**: Pulls the most relevant note from your notes and summarizes it
- **Chat**: Also does general chat. It guesses whether to give a general response or search, summarizes from your note. <br />
E.g *"how was your day?"* will give a general response. But *When did I go surfing?* should give a response from your notes
- **Note**: *Your query and top note from search result will be sent to OpenAI for processing*
#### Use
1. [Setup your OpenAI API key in Khoj](#set-your-openai-api-key-in-khoj)
2. Open [/chat?type=summarize](http://localhost:8000/chat?type=summarize)[^2]
3. Type your queries, see summarized response by Khoj from your notes
#### Demo
![](https://github.com/debanjum/khoj/blob/master/docs/khoj_chat_web_interface.png?)
### Use OpenAI Models for Search
#### Setup
1. Set `encoder-type`, `encoder` and `model-directory` under `asymmetric` and/or `symmetric` `search-type` in your `khoj.yml`[^1]:
```diff
asymmetric:
- encoder: "sentence-transformers/multi-qa-MiniLM-L6-cos-v1"
+ encoder: text-embedding-ada-002
+ encoder-type: src.utils.models.OpenAI
cross-encoder: "cross-encoder/ms-marco-MiniLM-L-6-v2"
- encoder-type: sentence_transformers.SentenceTransformer
- model_directory: "~/.khoj/search/asymmetric/"
+ model-directory: null
```
2. [Setup your OpenAI API key in Khoj](#set-your-openai-api-key-in-khoj)
3. Restart Khoj server to generate embeddings. It will take longer than with offline models.
#### Warnings
This configuration *uses an online model*
- It will **send all notes to OpenAI** to generate embeddings
- **All queries will be sent to OpenAI** when you search with Khoj
- You will be **charged by OpenAI** based on the total tokens processed
- It *requires an active internet connection* to search and index
## Miscellaneous
### Set your OpenAI API key in Khoj
If you want, Khoj can be configured to use OpenAI for search and chat.<br />
Add your OpenAI API to Khoj by using either of the two options below:
- Open the Khoj desktop GUI, add your [OpenAI API key](https://beta.openai.com/account/api-keys) and click *Configure*
Ensure khoj is started without the `--no-gui` flag. Check your system tray to see if Khoj 🦅 is minimized there.
- Set `openai-api-key` field under `processor.conversation` section in your `khoj.yml`[^1] to your [OpenAI API key](https://beta.openai.com/account/api-keys) and restart khoj:
```diff
processor:
conversation:
- openai-api-key: # "YOUR_OPENAI_API_KEY"
+ openai-api-key: sk-aaaaaaaaaaaaaaaaaaaaaaaahhhhhhhhhhhhhhhhhhhhhhhh
model: "text-davinci-003"
conversation-logfile: "~/.khoj/processor/conversation/conversation_logs.json"
```
**Warning**: *This will enable khoj to send your query and note(s) to OpenAI for processing*
### Beta API
- The beta [chat](http://localhost:8000/api/beta/chat), [summarize](http://localhost:8000/api/beta/summarize) and [search](http://localhost:8000/api/beta/search) API endpoints use [OpenAI API](https://openai.com/api/)
- They are disabled by default
- To use them:
1. [Setup your OpenAI API key in Khoj](#set-your-openai-api-key-in-khoj)
2. Interact with them from the [Khoj Swagger docs](http://locahost:8000/docs)[^2]
## Performance
### Query performance
- Semantic search using the bi-encoder is fairly fast at \<50 ms
- Reranking using the cross-encoder is slower at \<2s on 15 results. Tweak `top_k` to tradeoff speed for accuracy of results
- Filters in query (e.g by file, word or date) usually add \<20ms to query latency
### Indexing performance
- Indexing is more strongly impacted by the size of the source data
- Indexing 100K+ line corpus of notes takes about 10 minutes
- Indexing 4000+ images takes about 15 minutes and more than 8Gb of RAM
- Note: *It should only take this long on the first run* as the index is incrementally updated
### Miscellaneous
- Testing done on a Mac M1 and a \>100K line corpus of notes
- Search, indexing on a GPU has not been tested yet
## Development
### Visualize Codebase
*[Interactive Visualization](https://mango-dune-07a8b7110.1.azurestaticapps.net/?repo=debanjum%2Fkhoj)*
![](https://github.com/debanjum/khoj/blob/master/docs/khoj_codebase_visualization_0.2.1.png?)
### Setup
#### Using Pip
##### 1. Install
```shell
git clone https://github.com/debanjum/khoj && cd khoj
python3 -m venv .venv && source .venv/bin/activate
pip install -e .
```
##### 2. Configure
- Copy the `config/khoj_sample.yml` to `~/.khoj/khoj.yml`
- Set `input-files` or `input-filter` in each relevant `content-type` section of `~/.khoj/khoj.yml`
- Set `input-directories` field in `image` `content-type` section
- Delete `content-type` and `processor` sub-section(s) irrelevant for your use-case
##### 3. Run
```shell
khoj -vv
```
Load ML model, generate embeddings and expose API to query notes, images, transactions etc specified in config YAML
##### 4. Upgrade
```shell
# To Upgrade To Latest Stable Release
# Maps to the latest tagged version of khoj on master branch
pip install --upgrade khoj-assistant
# To Upgrade To Latest Pre-Release
# Maps to the latest commit on the master branch
pip install --upgrade --pre khoj-assistant
# To Upgrade To Specific Development Release.
# Useful to test, review a PR.
# Note: khoj-assistant is published to test PyPi on creating a PR
pip install -i https://test.pypi.org/simple/ khoj-assistant==0.1.5.dev57166025766
```
#### Using Docker
##### 1. Clone
```shell
git clone https://github.com/debanjum/khoj && cd khoj
```
##### 2. Configure
- **Required**: Update [docker-compose.yml](./docker-compose.yml) to mount your images, (org-mode or markdown) notes and beancount directories
- **Optional**: Edit application configuration in [khoj_docker.yml](./config/khoj_docker.yml)
##### 3. Run
```shell
docker-compose up -d
```
*Note: The first run will take time. Let it run, it\'s mostly not hung, just generating embeddings*
##### 4. Upgrade
```shell
docker-compose build --pull
```
#### Using Conda
##### 1. Install Dependencies
- [Install Conda](https://docs.conda.io/projects/conda/en/latest/user-guide/install/index.html)
##### 2. Install Khoj
```shell
git clone https://github.com/debanjum/khoj && cd khoj
conda env create -f config/environment.yml
conda activate khoj
python3 -m pip install pyqt6 # As conda does not support pyqt6 yet
```
##### 3. Configure
- Copy the `config/khoj_sample.yml` to `~/.khoj/khoj.yml`
- Set `input-files` or `input-filter` in each relevant `content-type` section of `~/.khoj/khoj.yml`
- Set `input-directories` field in `image` `content-type` section
- Delete `content-type`, `processor` sub-sections irrelevant for your use-case
##### 4. Run
```shell
python3 -m src.main -vv
```
Load ML model, generate embeddings and expose API to query notes, images, transactions etc specified in config YAML
##### 5. Upgrade
```shell
cd khoj
git pull origin master
conda deactivate khoj
conda env update -f config/environment.yml
conda activate khoj
```
### Test
```shell
pytest
```
## Credits
- [Multi-QA MiniLM Model](https://huggingface.co/sentence-transformers/multi-qa-MiniLM-L6-cos-v1), [All MiniLM Model](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) for Text Search. See [SBert Documentation](https://www.sbert.net/examples/applications/retrieve_rerank/README.html)
- [OpenAI CLIP Model](https://github.com/openai/CLIP) for Image Search. See [SBert Documentation](https://www.sbert.net/examples/applications/image-search/README.html)
- Charles Cave for [OrgNode Parser](http://members.optusnet.com.au/~charles57/GTD/orgnode.html)
- [Org.js](https://mooz.github.io/org-js/) to render Org-mode results on the Web interface
- [Markdown-it](https://github.com/markdown-it/markdown-it) to render Markdown results on the Web interface
[^1]: Default Khoj config file @ `~/.khoj/khoj.yml`
[^2]: Default Khoj url @ http://localhost:8000

2
config/environment.yml
View File

@@ -19,4 +19,4 @@ dependencies:
- aiofiles=0.8.0
- huggingface_hub=0.8.1
- dateparser=1.1.1
- schedule=1.1.0
- schedule=1.1.0

8
config/khoj_docker.yml
View File

@@ -4,20 +4,20 @@ content-type:
# If changing, the docker-compose volumes should also be changed to match.
org:
input-files: null
input-filter: ["/data/org/*.org"]
input-filter: ["/data/org/**/*.org"]
compressed-jsonl: "/data/embeddings/notes.jsonl.gz"
embeddings-file: "/data/embeddings/note_embeddings.pt"
index_heading_entries: false
markdown:
input-files: null
input-filter: ["/data/markdown/*.md"]
input-filter: ["/data/markdown/**/*.markdown"]
compressed-jsonl: "/data/embeddings/markdown.jsonl.gz"
embeddings-file: "/data/embeddings/markdown_embeddings.pt"
ledger:
input-files: null
input-filter: ["/data/ledger/*.beancount"]
input-filter: ["/data/ledger/**/*.beancount"]
compressed-jsonl: /data/embeddings/transactions.jsonl.gz
embeddings-file: /data/embeddings/transaction_embeddings.pt
@@ -52,4 +52,4 @@ processor:
#conversation:
# openai-api-key: null
# model: "text-davinci-003"
# conversation-logfile: "/data/embeddings/conversation_logs.json"
# conversation-logfile: "/data/embeddings/conversation_logs.json"

13
docker-compose.yml
View File

@@ -1,29 +1,28 @@
version: "3.9"
services:
server:
image: ghcr.io/debanjum/khoj:latest
image: ghcr.io/khoj-ai/khoj:latest
ports:
# If changing the local port (left hand side), no other changes required.
# If changing the remote port (right hand side),
# change the port in the args in the build section,
# If changing the remote port (right hand side),
# change the port in the args in the build section,
# as well as the port in the command section to match
- "8000:8000"
working_dir: /app
volumes:
- .:/app
# These mounted volumes hold the raw data that should be indexed for search.
# These mounted volumes hold the raw data that should be indexed for search.
# The path in your local directory (left hand side)
# points to the files you want to index.
# The path of the mounted directory (right hand side),
# must match the path prefix in your config file.
- ./tests/data/org/:/data/org/
- ./tests/data/images/:/data/images/
- ./tests/data/ledger/:/data/ledger/
- ./tests/data/music/:/data/music/
- ./tests/data/markdown/:/data/markdown/
- ./tests/data/pdf/:/data/pdf/
# Embeddings and models are populated after the first run
# You can set these volumes to point to empty directories on host
- ./tests/data/embeddings/:/data/embeddings/
- ./tests/data/models/:/data/models/
# Use 0.0.0.0 to explicitly set the host ip for the service on the container. https://pythonspeed.com/articles/docker-connection-refused/
command: --no-gui --host="0.0.0.0" --port=8000 -c=config/khoj_docker.yml -vv
command: --host="0.0.0.0" --port=8000 -c=config/khoj_docker.yml -vv

BIN
docs/khoj_chat_on_emacs_0.5.0.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 302 KiB

16
manifest.json
View File

@@ -1,10 +1,10 @@
{
"id": "khoj",
"name": "Khoj",
"version": "0.2.1",
"minAppVersion": "0.15.0",
"description": "Natural, Incremental Search for your Second Brain 🦅",
"author": "Debanjum Singh Solanky",
"authorUrl": "https://github.com/debanjum",
"isDesktopOnly": false
"id": "khoj",
"name": "Khoj",
"version": "0.8.0",
"minAppVersion": "0.15.0",
"description": "An AI Personal Assistant for your Digital Brain",
"author": "Debanjum Singh Solanky",
"authorUrl": "https://github.com/debanjum",
"isDesktopOnly": false
}

114
pyproject.toml Normal file
View File

@@ -0,0 +1,114 @@
[build-system]
requires = ["hatchling", "hatch-vcs"]
build-backend = "hatchling.build"
[project]
name = "khoj-assistant"
description = "An AI personal assistant for your Digital Brain"
readme = "README.md"
license = "GPL-3.0-or-later"
requires-python = ">=3.8"
authors = [
{ name = "Debanjum Singh Solanky, Saba Imran" },
]
keywords = [
"search",
"semantic-search",
"productivity",
"NLP",
"AI",
"org-mode",
"markdown",
"images",
"pdf",
]
classifiers = [
"Development Status :: 4 - Beta",
"License :: OSI Approved :: GNU General Public License v3 (GPLv3)",
"Operating System :: OS Independent",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Topic :: Internet :: WWW/HTTP :: Indexing/Search",
"Topic :: Scientific/Engineering :: Artificial Intelligence",
"Topic :: Scientific/Engineering :: Human Machine Interfaces",
"Topic :: Text Processing :: Linguistic",
]
dependencies = [
"dateparser == 1.1.1",
"defusedxml == 0.7.1",
"fastapi == 0.77.1",
"jinja2 == 3.1.2",
"openai >= 0.27.0",
"tiktoken >= 0.3.0",
"tenacity >= 8.2.2",
"pillow == 9.3.0",
"pydantic >= 1.10.10",
"pyqt6 == 6.3.1",
"pyyaml == 6.0",
"rich >= 13.3.1",
"schedule == 1.1.0",
"sentence-transformers == 2.2.2",
"torch >= 2.0.1",
"uvicorn == 0.17.6",
"aiohttp == 3.8.4",
"langchain >= 0.0.187",
"pypdf >= 3.9.0",
"requests >= 2.26.0",
"bs4 >= 0.0.1",
]
dynamic = ["version"]
[project.urls]
Homepage = "https://github.com/khoj-ai/khoj#readme"
Issues = "https://github.com/khoj-ai/khoj/issues"
Discussions = "https://github.com/khoj-ai/khoj/discussions"
Releases = "https://github.com/khoj-ai/khoj/releases"
[project.scripts]
khoj = "khoj.main:run"
[project.optional-dependencies]
test = [
"pytest >= 7.1.2",
"freezegun >= 1.2.0",
"factory-boy >= 3.2.1",
"trio >= 0.22.0",
]
dev = [
"khoj-assistant[test]",
"mypy >= 1.0.1",
"black >= 23.1.0",
"pre-commit >= 3.0.4",
]
[tool.hatch.version]
source = "vcs"
raw-options.local_scheme = "no-local-version" # PEP440 compliant version for PyPi
[tool.hatch.build.targets.sdist]
include = ["src/khoj"]
[tool.hatch.build.targets.wheel]
packages = ["src/khoj"]
[tool.mypy]
files = "src/khoj"
pretty = true
strict_optional = false
install_types = true
ignore_missing_imports = true
non_interactive = true
show_error_codes = true
warn_unused_ignores = false
[tool.black]
line-length = 120
[tool.pytest.ini_options]
addopts = "--strict-markers"
markers = [
"chatquality: Evaluate chatbot capabilities and quality",
]

82
scripts/bump_version.sh Executable file
View File

@@ -0,0 +1,82 @@
#!/bin/zsh
project_root=$PWD
while getopts 'nc:' opt;
do
case "${opt}" in
c)
# Get current project version
current_version=$OPTARG
# Bump Obsidian plugin to current version
cd $project_root/src/interface/obsidian
sed -E -i.bak "s/version\": \"(.*)\",/version\": \"$current_version\",/" package.json
sed -E -i.bak "s/version\": \"(.*)\"/version\": \"$current_version\"/" manifest.json
cp $project_root/versions.json .
npm run version # append current version
rm *.bak
# Bump Emacs package to current version
cd ../emacs
sed -E -i.bak "s/^;; Version: (.*)/;; Version: $current_version/" khoj.el
git add khoj.el
rm *.bak
# Copy current obsidian versioned files to project root
cd $project_root
cp src/interface/obsidian/versions.json .
cp src/interface/obsidian/manifest.json .
# Run pre-commit validation to fix jsons
pre-commit run --hook-stage manual --all
# Commit changes and tag commit for release
git add \
$project_root/src/interface/obsidian/package.json \
$project_root/src/interface/obsidian/manifest.json \
$project_root/src/interface/obsidian/versions.json \
$project_root/src/interface/emacs/khoj.el \
$project_root/manifest.json \
$project_root/versions.json
git commit -m "Release Khoj version $current_version"
git tag $current_version master
;;
n)
# Induce hatch to compute next version number
# remove .dev[commits-since-tag] version suffix from hatch computed version number
next_version=$(touch bump.txt && git add bump.txt && hatch version | sed 's/\.dev.*//g')
git rm --cached -- bump.txt && rm bump.txt
# Bump Obsidian plugins to next version
cd $project_root/src/interface/obsidian
sed -E -i.bak "s/version\": \"(.*)\",/version\": \"$next_version\",/" package.json
sed -E -i.bak "s/version\": \"(.*)\"/version\": \"$next_version\"/" manifest.json
npm run version # updates versions.json
rm *.bak
# Bump Emacs package to next version
cd $project_root/src/interface/emacs
sed -E -i.bak "s/^;; Version: (.*)/;; Version: $next_version/" khoj.el
rm *.bak
# Run pre-commit validations to fix jsons
pre-commit run --hook-stage manual --all
# Commit changes
git add \
$project_root/src/interface/obsidian/package.json \
$project_root/src/interface/obsidian/manifest.json \
$project_root/src/interface/obsidian/versions.json \
$project_root/src/interface/emacs/khoj.el
git commit -m "Bump Khoj to pre-release version $next_version"
;;
?)
echo -e "Invalid command option.\nUsage: $(basename $0) [-c] [-n]"
exit 1
;;
esac
done
# Restore State
cd $project_root

57
setup.py
View File

@@ -1,57 +0,0 @@
#!/usr/bin/env python
from setuptools import find_packages, setup
from pathlib import Path
this_directory = Path(__file__).parent
setup(
name='khoj-assistant',
version='0.2.5',
description="A natural language search engine for your personal notes, transactions and images",
long_description=(this_directory / "Readme.md").read_text(encoding="utf-8"),
long_description_content_type="text/markdown",
author='Debanjum Singh Solanky, Saba Imran',
author_email='debanjum+pypi@gmail.com, narmiabas@gmail.com',
url='https://github.com/debanjum/khoj',
license="GPLv3",
keywords="search semantic-search productivity NLP org-mode markdown beancount images",
python_requires=">=3.8, <4",
packages=find_packages(
where=".",
exclude=["tests*"],
include=["src*"]
),
install_requires=[
"numpy == 1.22.4",
"torch == 1.13.1",
"torchvision == 0.14.1",
"transformers == 4.21.0",
"sentence-transformers == 2.1.0",
"openai == 0.20.0",
"huggingface_hub == 0.8.1",
"pydantic == 1.9.1",
"fastapi == 0.77.1",
"uvicorn == 0.17.6",
"jinja2 == 3.1.2",
"pyyaml == 6.0",
"pytest == 7.1.2",
"pillow == 9.3.0",
"aiofiles == 0.8.0",
"dateparser == 1.1.1",
"pyqt6 == 6.3.1",
"defusedxml == 0.7.1",
'schedule == 1.1.0',
],
include_package_data=True,
entry_points={"console_scripts": ["khoj = src.main:run"]},
classifiers=[
"Development Status :: 4 - Beta",
"License :: OSI Approved :: GNU General Public License v3 (GPLv3)",
"Operating System :: OS Independent",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
]
)

136
src/configure.py
View File

@@ -1,136 +0,0 @@
# Standard Packages
import sys
import logging
import json
# External Packages
import schedule
# Internal Packages
from src.processor.ledger.beancount_to_jsonl import BeancountToJsonl
from src.processor.markdown.markdown_to_jsonl import MarkdownToJsonl
from src.processor.org_mode.org_to_jsonl import OrgToJsonl
from src.search_type import image_search, text_search
from src.utils.config import SearchType, SearchModels, ProcessorConfigModel, ConversationProcessorConfigModel
from src.utils import state
from src.utils.helpers import LRU, resolve_absolute_path
from src.utils.rawconfig import FullConfig, ProcessorConfig
from src.search_filter.date_filter import DateFilter
from src.search_filter.word_filter import WordFilter
from src.search_filter.file_filter import FileFilter
logger = logging.getLogger(__name__)
def configure_server(args, required=False):
if args.config is None:
if required:
logger.error(f'Exiting as Khoj is not configured.\nConfigure it via GUI or by editing {state.config_file}.')
sys.exit(1)
else:
logger.warn(f'Khoj is not configured.\nConfigure it via khoj GUI, plugins or by editing {state.config_file}.')
return
else:
state.config = args.config
# Initialize Processor from Config
state.processor_config = configure_processor(args.config.processor)
# Initialize the search model from Config
state.search_index_lock.acquire()
state.model = configure_search(state.model, state.config, args.regenerate)
state.search_index_lock.release()
@schedule.repeat(schedule.every(1).hour)
def update_search_index():
state.search_index_lock.acquire()
state.model = configure_search(state.model, state.config, regenerate=False)
state.search_index_lock.release()
logger.info("Search Index updated via Scheduler")
def configure_search(model: SearchModels, config: FullConfig, regenerate: bool, t: SearchType = None):
# Initialize Org Notes Search
if (t == SearchType.Org or t == None) and config.content_type.org:
# Extract Entries, Generate Notes Embeddings
model.orgmode_search = text_search.setup(
OrgToJsonl,
config.content_type.org,
search_config=config.search_type.asymmetric,
regenerate=regenerate,
filters=[DateFilter(), WordFilter(), FileFilter()])
# Initialize Org Music Search
if (t == SearchType.Music or t == None) and config.content_type.music:
# Extract Entries, Generate Music Embeddings
model.music_search = text_search.setup(
OrgToJsonl,
config.content_type.music,
search_config=config.search_type.asymmetric,
regenerate=regenerate,
filters=[DateFilter(), WordFilter()])
# Initialize Markdown Search
if (t == SearchType.Markdown or t == None) and config.content_type.markdown:
# Extract Entries, Generate Markdown Embeddings
model.markdown_search = text_search.setup(
MarkdownToJsonl,
config.content_type.markdown,
search_config=config.search_type.asymmetric,
regenerate=regenerate,
filters=[DateFilter(), WordFilter(), FileFilter()])
# Initialize Ledger Search
if (t == SearchType.Ledger or t == None) and config.content_type.ledger:
# Extract Entries, Generate Ledger Embeddings
model.ledger_search = text_search.setup(
BeancountToJsonl,
config.content_type.ledger,
search_config=config.search_type.symmetric,
regenerate=regenerate,
filters=[DateFilter(), WordFilter(), FileFilter()])
# Initialize Image Search
if (t == SearchType.Image or t == None) and config.content_type.image:
# Extract Entries, Generate Image Embeddings
model.image_search = image_search.setup(
config.content_type.image,
search_config=config.search_type.image,
regenerate=regenerate)
# Invalidate Query Cache
state.query_cache = LRU()
return model
def configure_processor(processor_config: ProcessorConfig):
if not processor_config:
return
processor = ProcessorConfigModel()
# Initialize Conversation Processor
if processor_config.conversation:
processor.conversation = configure_conversation_processor(processor_config.conversation)
return processor
def configure_conversation_processor(conversation_processor_config):
conversation_processor = ConversationProcessorConfigModel(conversation_processor_config)
conversation_logfile = resolve_absolute_path(conversation_processor.conversation_logfile)
if conversation_logfile.is_file():
# Load Metadata Logs from Conversation Logfile
with conversation_logfile.open('r') as f:
conversation_processor.meta_log = json.load(f)
logger.info('Conversation logs loaded from disk.')
else:
# Initialize Conversation Logs
conversation_processor.meta_log = {}
conversation_processor.chat_session = ""
return conversation_processor

72
src/interface/desktop/file_browser.py
View File

@@ -1,72 +0,0 @@
# External Packages
from PyQt6 import QtWidgets
from PyQt6.QtCore import QDir
# Internal Packages
from src.utils.config import SearchType
from src.utils.helpers import is_none_or_empty
class FileBrowser(QtWidgets.QWidget):
def __init__(self, title, search_type: SearchType=None, default_files:list=[]):
QtWidgets.QWidget.__init__(self)
layout = QtWidgets.QHBoxLayout()
self.setLayout(layout)
self.search_type = search_type
self.filter_name = self.getFileFilter(search_type)
self.dirpath = QDir.homePath()
self.label = QtWidgets.QLabel()
self.label.setText(title)
self.label.setFixedWidth(95)
self.label.setWordWrap(True)
layout.addWidget(self.label)
self.lineEdit = QtWidgets.QPlainTextEdit(self)
self.lineEdit.setFixedWidth(330)
self.setFiles(default_files)
self.lineEdit.setFixedHeight(min(7+20*len(self.lineEdit.toPlainText().split('\n')),90))
self.lineEdit.textChanged.connect(self.updateFieldHeight)
layout.addWidget(self.lineEdit)
self.button = QtWidgets.QPushButton('Add')
self.button.clicked.connect(self.storeFilesSelectedInFileDialog)
layout.addWidget(self.button)
layout.addStretch()
def getFileFilter(self, search_type):
if search_type == SearchType.Org:
return 'Org-Mode Files (*.org)'
elif search_type == SearchType.Ledger:
return 'Beancount Files (*.bean *.beancount)'
elif search_type == SearchType.Markdown:
return 'Markdown Files (*.md *.markdown)'
elif search_type == SearchType.Music:
return 'Org-Music Files (*.org)'
elif search_type == SearchType.Image:
return 'Images (*.jp[e]g)'
def storeFilesSelectedInFileDialog(self):
filepaths = self.getPaths()
if self.search_type == SearchType.Image:
filepaths.append(QtWidgets.QFileDialog.getExistingDirectory(self, caption='Choose Folder',
directory=self.dirpath))
else:
filepaths.extend(QtWidgets.QFileDialog.getOpenFileNames(self, caption='Choose Files',
directory=self.dirpath,
filter=self.filter_name)[0])
self.setFiles(filepaths)
def setFiles(self, paths:list):
self.filepaths = [path for path in paths if not is_none_or_empty(path)]
self.lineEdit.setPlainText("\n".join(self.filepaths))
def getPaths(self) -> list:
if self.lineEdit.toPlainText() == '':
return []
else:
return self.lineEdit.toPlainText().split('\n')
def updateFieldHeight(self):
self.lineEdit.setFixedHeight(min(7+20*len(self.lineEdit.toPlainText().split('\n')),90))

27
src/interface/desktop/labelled_text_field.py
View File

@@ -1,27 +0,0 @@
# External Packages
from PyQt6 import QtWidgets
# Internal Packages
from src.utils.config import ProcessorType
class LabelledTextField(QtWidgets.QWidget):
def __init__(self, title, processor_type: ProcessorType=None, default_value: str=None):
QtWidgets.QWidget.__init__(self)
layout = QtWidgets.QHBoxLayout()
self.setLayout(layout)
self.processor_type = processor_type
self.label = QtWidgets.QLabel()
self.label.setText(title)
self.label.setFixedWidth(95)
self.label.setWordWrap(True)
layout.addWidget(self.label)
self.input_field = QtWidgets.QTextEdit(self)
self.input_field.setFixedWidth(410)
self.input_field.setFixedHeight(27)
self.input_field.setText(default_value)
layout.addWidget(self.input_field)
layout.addStretch()

318
src/interface/desktop/main_window.py
View File

@@ -1,318 +0,0 @@
# Standard Packages
from enum import Enum
from pathlib import Path
from copy import deepcopy
import webbrowser
# External Packages
from PyQt6 import QtGui, QtWidgets
from PyQt6.QtCore import Qt, QThread, QObject, pyqtSignal
# Internal Packages
from src.configure import configure_server
from src.interface.desktop.file_browser import FileBrowser
from src.interface.desktop.labelled_text_field import LabelledTextField
from src.utils import constants, state, yaml as yaml_utils
from src.utils.cli import cli
from src.utils.config import SearchType, ProcessorType
from src.utils.helpers import merge_dicts, resolve_absolute_path
class MainWindow(QtWidgets.QMainWindow):
"""Create Window to Configure Khoj
Allow user to
1. Configure content types to search
2. Configure conversation processor
3. Save the configuration to khoj.yml
"""
def __init__(self, config_file: Path):
super(MainWindow, self).__init__()
self.config_file = config_file
# Set regenerate flag to regenerate embeddings everytime user clicks configure
if state.cli_args:
state.cli_args += ['--regenerate']
else:
state.cli_args = ['--regenerate']
# Load config from existing config, if exists, else load from default config
if resolve_absolute_path(self.config_file).exists():
self.first_run = False
self.current_config = yaml_utils.load_config_from_file(self.config_file)
else:
self.first_run = True
self.current_config = deepcopy(constants.default_config)
self.new_config = self.current_config
# Initialize Configure Window
self.setWindowTitle("Khoj")
self.setFixedWidth(600)
# Set Window Icon
icon_path = constants.web_directory / 'assets/icons/favicon-144x144.png'
self.setWindowIcon(QtGui.QIcon(f'{icon_path.absolute()}'))
# Initialize Configure Window Layout
self.layout = QtWidgets.QVBoxLayout()
# Add Settings Panels for each Search Type to Configure Window Layout
self.search_settings_panels = []
for search_type in SearchType:
current_content_config = self.current_config['content-type'].get(search_type, {})
self.search_settings_panels += [self.add_settings_panel(current_content_config, search_type)]
# Add Conversation Processor Panel to Configure Screen
self.processor_settings_panels = []
conversation_type = ProcessorType.Conversation
current_conversation_config = self.current_config['processor'].get(conversation_type, {})
self.processor_settings_panels += [self.add_processor_panel(current_conversation_config, conversation_type)]
# Add Action Buttons Panel
self.add_action_panel()
# Set the central widget of the Window. Widget will expand
# to take up all the space in the window by default.
self.config_window = QtWidgets.QWidget()
self.config_window.setLayout(self.layout)
self.setCentralWidget(self.config_window)
self.position_window()
def add_settings_panel(self, current_content_config: dict, search_type: SearchType):
"Add Settings Panel for specified Search Type. Toggle Editable Search Types"
# Get current files from config for given search type
if search_type == SearchType.Image:
current_content_files = current_content_config.get('input-directories', [])
file_input_text = f'{search_type.name} Folders'
else:
current_content_files = current_content_config.get('input-files', [])
file_input_text = f'{search_type.name} Files'
# Create widgets to display settings for given search type
search_type_settings = QtWidgets.QWidget()
search_type_layout = QtWidgets.QVBoxLayout(search_type_settings)
enable_search_type = SearchCheckBox(f"Search {search_type.name}", search_type)
# Add file browser to set input files for given search type
input_files = FileBrowser(file_input_text, search_type, current_content_files or [])
# Set enabled/disabled based on checkbox state
enable_search_type.setChecked(current_content_files is not None and len(current_content_files) > 0)
input_files.setEnabled(enable_search_type.isChecked())
enable_search_type.stateChanged.connect(lambda _: input_files.setEnabled(enable_search_type.isChecked()))
# Add setting widgets for given search type to panel
search_type_layout.addWidget(enable_search_type)
search_type_layout.addWidget(input_files)
self.layout.addWidget(search_type_settings)
return search_type_settings
def add_processor_panel(self, current_conversation_config: dict, processor_type: ProcessorType):
"Add Conversation Processor Panel"
# Get current settings from config for given processor type
current_openai_api_key = current_conversation_config.get('openai-api-key', None)
# Create widgets to display settings for given processor type
processor_type_settings = QtWidgets.QWidget()
processor_type_layout = QtWidgets.QVBoxLayout(processor_type_settings)
enable_conversation = ProcessorCheckBox(f"Conversation", processor_type)
# Add file browser to set input files for given processor type
input_field = LabelledTextField("OpenAI API Key", processor_type, current_openai_api_key)
# Set enabled/disabled based on checkbox state
enable_conversation.setChecked(current_openai_api_key is not None)
input_field.setEnabled(enable_conversation.isChecked())
enable_conversation.stateChanged.connect(lambda _: input_field.setEnabled(enable_conversation.isChecked()))
# Add setting widgets for given processor type to panel
processor_type_layout.addWidget(enable_conversation)
processor_type_layout.addWidget(input_field)
self.layout.addWidget(processor_type_settings)
return processor_type_settings
def add_action_panel(self):
"Add Action Panel"
# Button to Save Settings
action_bar = QtWidgets.QWidget()
action_bar_layout = QtWidgets.QHBoxLayout(action_bar)
self.configure_button = QtWidgets.QPushButton("Configure", clicked=self.configure_app)
self.search_button = QtWidgets.QPushButton("Search", clicked=lambda: webbrowser.open(f'http://{state.host}:{state.port}/'))
self.search_button.setEnabled(not self.first_run)
action_bar_layout.addWidget(self.configure_button)
action_bar_layout.addWidget(self.search_button)
self.layout.addWidget(action_bar)
def get_default_config(self, search_type:SearchType=None, processor_type:ProcessorType=None):
"Get default config"
config = constants.default_config
if search_type:
return config['content-type'][search_type]
elif processor_type:
return config['processor'][processor_type]
else:
return config
def add_error_message(self, message: str):
"Add Error Message to Configure Screen"
# Remove any existing error messages
for message_prefix in ErrorType:
for i in reversed(range(self.layout.count())):
current_widget = self.layout.itemAt(i).widget()
if isinstance(current_widget, QtWidgets.QLabel) and current_widget.text().startswith(message_prefix.value):
self.layout.removeWidget(current_widget)
current_widget.deleteLater()
# Add new error message
if message:
error_message = QtWidgets.QLabel()
error_message.setWordWrap(True)
error_message.setText(message)
error_message.setStyleSheet("color: red")
self.layout.addWidget(error_message)
def update_search_settings(self):
"Update config with search settings from UI"
for settings_panel in self.search_settings_panels:
for child in settings_panel.children():
if not isinstance(child, (SearchCheckBox, FileBrowser)):
continue
if isinstance(child, SearchCheckBox):
# Search Type Disabled
if not child.isChecked() and child.search_type in self.new_config['content-type']:
del self.new_config['content-type'][child.search_type]
# Search Type (re)-Enabled
if child.isChecked():
current_search_config = self.current_config['content-type'].get(child.search_type, {})
default_search_config = self.get_default_config(search_type = child.search_type)
self.new_config['content-type'][child.search_type.value] = merge_dicts(current_search_config, default_search_config)
elif isinstance(child, FileBrowser) and child.search_type in self.new_config['content-type']:
if child.search_type.value == SearchType.Image:
self.new_config['content-type'][child.search_type.value]['input-directories'] = child.getPaths() if child.getPaths() != [] else None
else:
self.new_config['content-type'][child.search_type.value]['input-files'] = child.getPaths() if child.getPaths() != [] else None
def update_processor_settings(self):
"Update config with conversation settings from UI"
for settings_panel in self.processor_settings_panels:
for child in settings_panel.children():
if not isinstance(child, (ProcessorCheckBox, LabelledTextField)):
continue
if isinstance(child, ProcessorCheckBox):
# Processor Type Disabled
if not child.isChecked() and child.processor_type in self.new_config['processor']:
del self.new_config['processor'][child.processor_type]
# Processor Type (re)-Enabled
if child.isChecked():
current_processor_config = self.current_config['processor'].get(child.processor_type, {})
default_processor_config = self.get_default_config(processor_type = child.processor_type)
self.new_config['processor'][child.processor_type.value] = merge_dicts(current_processor_config, default_processor_config)
elif isinstance(child, LabelledTextField) and child.processor_type in self.new_config['processor']:
if child.processor_type == ProcessorType.Conversation:
self.new_config['processor'][child.processor_type.value]['openai-api-key'] = child.input_field.toPlainText() if child.input_field.toPlainText() != '' else None
def save_settings_to_file(self) -> bool:
"Save validated settings to file"
# Validate config before writing to file
try:
yaml_utils.parse_config_from_string(self.new_config)
except Exception as e:
print(f"Error validating config: {e}")
self.add_error_message(f"{ErrorType.ConfigValidationError.value}: {e}")
return False
# Save the config to app config file
self.add_error_message(None)
yaml_utils.save_config_to_file(self.new_config, self.config_file)
return True
def load_updated_settings(self):
"Hot swap to use the updated config from config file"
# Load parsed, validated config from app config file
args = cli(state.cli_args)
self.current_config = self.new_config
# Configure server with loaded config
configure_server(args, required=True)
def configure_app(self):
"Save the new settings to khoj.yml. Reload app with updated settings"
self.update_search_settings()
self.update_processor_settings()
if self.save_settings_to_file():
# Setup thread to load updated settings in background
self.thread = QThread()
self.settings_loader = SettingsLoader(self.load_updated_settings)
self.settings_loader.moveToThread(self.thread)
# Connect slots and signals for thread
self.thread.started.connect(self.settings_loader.run)
self.settings_loader.finished.connect(self.thread.quit)
self.settings_loader.finished.connect(self.settings_loader.deleteLater)
self.settings_loader.error.connect(self.add_error_message)
self.thread.finished.connect(self.thread.deleteLater)
# Start thread
self.thread.start()
# Disable Save Button
self.search_button.setEnabled(False)
self.configure_button.setEnabled(False)
self.configure_button.setText("Configuring...")
# Reset UI
self.thread.finished.connect(lambda: self.configure_button.setText("Configure"))
self.thread.finished.connect(lambda: self.configure_button.setEnabled(True))
self.thread.finished.connect(lambda: self.search_button.setEnabled(True))
def position_window(self):
"Position the window at center of X axis and near top on Y axis"
window_rectangle = self.geometry()
screen_center = self.screen().availableGeometry().center()
window_rectangle.moveCenter(screen_center)
self.move(window_rectangle.topLeft().x(), 25)
def show_on_top(self):
"Bring Window on Top"
self.show()
self.setWindowState(Qt.WindowState.WindowActive)
self.activateWindow() # For Bringing to Top on Windows
self.raise_() # For Bringing to Top from Minimized State on OSX
class SettingsLoader(QObject):
"Load Settings Thread"
finished = pyqtSignal()
error = pyqtSignal(str)
def __init__(self, load_settings_func):
super(SettingsLoader, self).__init__()
self.load_settings_func = load_settings_func
def run(self):
"Load Settings"
try:
self.load_settings_func()
except FileNotFoundError as e:
self.error.emit(f"{ErrorType.ConfigLoadingError.value}: {e}")
else:
self.error.emit(None)
self.finished.emit()
class SearchCheckBox(QtWidgets.QCheckBox):
def __init__(self, text, search_type: SearchType, parent=None):
self.search_type = search_type
super(SearchCheckBox, self).__init__(text, parent=parent)
class ProcessorCheckBox(QtWidgets.QCheckBox):
def __init__(self, text, processor_type: ProcessorType, parent=None):
self.processor_type = processor_type
super(ProcessorCheckBox, self).__init__(text, parent=parent)
class ErrorType(Enum):
"Error Types"
ConfigLoadingError = "Config Loading Error"
ConfigValidationError = "Config Validation Error"

135
src/interface/emacs/README.org
View File

@@ -1,65 +1,101 @@
* Khoj Emacs 🦅
[[https://stable.melpa.org/#/khoj][file:https://stable.melpa.org/packages/khoj-badge.svg]] [[https://melpa.org/#/khoj][file:https://melpa.org/packages/khoj-badge.svg]] [[https://github.com/debanjum/khoj/actions/workflows/build_khoj_el.yml][https://github.com/debanjum/khoj/actions/workflows/build_khoj_el.yml/badge.svg?]]
/Natural language search from within Emacs using [[https://github.com/debanjum/khoj][Khoj]]/
[[https://github.com/khoj-ai/khoj/edit/master/src/interface/emacs/README.org][file:/src/khoj/interface/web/assets/icons/khoj-logo-sideways-200.png]] Emacs
[[https://stable.melpa.org/#/khoj][file:https://stable.melpa.org/packages/khoj-badge.svg]] [[https://melpa.org/#/khoj][file:https://melpa.org/packages/khoj-badge.svg]] [[https://github.com/khoj-ai/khoj/actions/workflows/build_khoj_el.yml][https://github.com/khoj-ai/khoj/actions/workflows/build_khoj_el.yml/badge.svg?]] [[https://github.com/khoj-ai/khoj/actions/workflows/test_khoj_el.yml][https://github.com/khoj-ai/khoj/actions/workflows/test_khoj_el.yml/badge.svg?]]
/An AI personal assistant for your digital brain/
** Table of Contents
- [[https://github.com/debanjum/khoj/tree/master/src/interface/emacs#features][Features]]
- [[https://github.com/debanjum/khoj/tree/master/src/interface/emacs#Interface][Interface]]
- [[https://github.com/debanjum/khoj/tree/master/src/interface/emacs#Setup][Setup]]
- [[https://github.com/debanjum/khoj/tree/master/src/interface/emacs#1-Setup-Backend][Setup Backend]]
- [[https://github.com/debanjum/khoj/tree/master/src/interface/emacs#2-Install-Khojel][Install Khoj.el]]
- [[https://github.com/debanjum/khoj/tree/master/src/interface/emacs#Use][Use]]
- [[https://github.com/debanjum/khoj/tree/master/src/interface/emacs#Search][Search]]
- [[https://github.com/debanjum/khoj/tree/master/src/interface/emacs#Find-similar-entries][Find Similar Entries]]
- [[https://github.com/debanjum/khoj/tree/master/src/interface/emacs#Advanced-usage][Advanced Usage]]
- [[https://github.com/debanjum/khoj/tree/master/src/interface/emacs#Khoj-menu][Khoj Menu]]
- [[https://github.com/debanjum/khoj/tree/master/src/interface/emacs#Upgrade][Upgrade]]
- [[https://github.com/debanjum/khoj/tree/master/src/interface/emacs#Upgrade-Khoj-Backend][Upgrade Backend]]
- [[https://github.com/debanjum/khoj/tree/master/src/interface/emacs#Upgrade-Khojel][Upgrade Khoj.el]]
- [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#features][Features]]
- [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Interface][Interface]]
- [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Setup][Setup]]
- [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Direct-Install][Direct Install]]
- [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Minimal-Install][Minimal Install]]
- [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Standard-Install][Standard Install]]
- [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#With-Straight.el][With Straight.el]]
- [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Use][Use]]
- [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Search][Search]]
- [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Chat][Chat]]
- [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Find-similar-entries][Find Similar Entries]]
- [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Advanced-usage][Advanced Usage]]
- [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Khoj-menu][Khoj Menu]]
- [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Upgrade][Upgrade]]
- [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Upgrade-Khoj-Backend][Upgrade Backend]]
- [[https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs#Upgrade-Khojel][Upgrade Khoj.el]]
** Features
- *Natural*: Advanced natural language understanding using Transformer based ML Models
- *Local*: Your personal data stays local. All search, indexing is done on your machine*
- *Incremental*: Incremental search for a fast, search-as-you-type experience
- *Search*
- *Natural*: Advanced natural language understanding using Transformer based ML Models
- *Local*: Your personal data stays local. All search, indexing is done on your machine*
- *Incremental*: Incremental search for a fast, search-as-you-type experience
- *Chat*
- *Faster answers*: Find answers faster than search
- *Iterative discovery*: Iteratively explore and (re-)discover your notes
- *Assisted creativity*: Smoothly weave across answer retrieval and content generation
** Interface
*** Search UI
[[/docs/khoj_on_emacs.png]]
*** Chat UI
[[/docs/khoj_chat_on_emacs_0.5.0.png]]
** Setup
*** 1. Setup Backend
#+begin_src shell
pip install khoj-assistant && khoj
#+end_src
- /Make sure [[https://realpython.com/installing-python/][python]] and [[https://pip.pypa.io/en/stable/installation/][pip]] are installed on your machine/
*** 2. Install Khoj.el
**** Using MELPA
#+begin_src elisp
- /khoj.el attempts to automatically install, start and configure the khoj server./
If this fails, follow [[https://github.com/khoj-ai/khoj/tree/master/#Setup][these instructions]] to manually setup the khoj server.
*** Direct Install
#+begin_src elisp
M-x package-install khoj
#+end_src elisp
#+end_src
Add below snippet to your Emacs config file
#+begin_src elisp
;; Install Khoj Package from MELPA Stable
(use-package khoj
:ensure t
:pin melpa-stable
:bind ("C-c s" . 'khoj))
#+end_src
*** Minimal Install
Add below snippet to your Emacs config file.
Indexes your org-agenda files, by default.
Note: Install ~khoj.el~ from MELPA (instead of MELPA Stable) if you installed the pre-release version of khoj
- That is, use ~:pin melpa~ to install khoj.el in above snippet if khoj was installed with ~pip install --pre khoj-assistant~
- Else use ~:pin melpa-stable~ to install khoj.el in above snippet if khoj was installed with ~pip install khoj-assistant~
- This ensures both khoj.el and khoj app are from the same version (tagged or latest)
#+begin_src elisp
;; Install Khoj Package from MELPA Stable
(use-package khoj
:ensure t
:pin melpa-stable
:bind ("C-c s" . 'khoj)
#+end_src
- Note: Install ~khoj.el~ from MELPA (instead of MELPA Stable) if you installed the pre-release version of khoj
- That is, use ~:pin melpa~ to install khoj.el in above snippet if khoj server was installed with ~--pre~ flag, i.e ~pip install --pre khoj-assistant~
- Else use ~:pin melpa-stable~ to install khoj.el in above snippet if khoj was installed with ~pip install khoj-assistant~
- This ensures both khoj.el and khoj app are from the same version (git tagged or latest)
*** Standard Install
Add below snippet to your Emacs config file.
Indexes the specified org files, directories. Sets up OpenAI API key for Khoj Chat
#+begin_src elisp
;; Install Khoj Package from MELPA Stable
(use-package khoj
:ensure t
:pin melpa-stable
:bind ("C-c s" . 'khoj)
:config (setq khoj-org-directories '("~/docs/org-roam" "~/docs/notes")
khoj-org-files '("~/docs/todo.org" "~/docs/work.org")
khoj-openai-api-key "YOUR_OPENAI_API_KEY")) ; required to enable chat
#+end_src
*** With [[https://github.com/raxod502/straight.el][Straight.el]]
Add below snippet to your Emacs config file.
Indexes the specified org files, directories. Sets up OpenAI API key for Khoj Chat
**** Using [[https://github.com/raxod502/straight.el][Straight.el]]
Add below snippet to your Emacs config file
#+begin_src elisp
;; Install Khoj Package using Straight.el
(use-package khoj
:after org
:straight (khoj :type git :host github :repo "debanjum/khoj" :files (:defaults "src/interface/emacs/khoj.el"))
:bind ("C-c s" . 'khoj))
:straight (khoj :type git :host github :repo "khoj-ai/khoj" :files (:defaults "src/interface/emacs/khoj.el"))
:bind ("C-c s" . 'khoj)
:config (setq khoj-org-directories '("~/docs/org-roam" "~/docs/notes")
khoj-org-files '("~/docs/todo.org" "~/docs/work.org")
khoj-openai-api-key "YOUR_OPENAI_API_KEY" ; required to enable chat)
#+end_src
** Use
@@ -70,13 +106,22 @@
e.g "What is the meaning of life?", "My life goals for 2023"
*** Chat
1. Hit ~C-c s c~ (or ~M-x khoj RET c~) to open khoj chat
2. Ask questions in a natural, conversational style
E.g "When did I file my taxes last year?"
See [[https://github.com/khoj-ai/khoj/tree/master/#Khoj-Chat][Khoj Chat]] for more details
*** Find Similar Entries
This feature finds entries similar to the one you are currently on.
1. Move cursor to the org-mode entry, markdown section or text paragraph you want to find similar entries for
2. Hit ~C-c s f~ (or ~M-x khoj RET f~) to find similar entries
*** Advanced Usage
- Add [[https://github.com/debanjum/khoj/#query-filters][query filters]] during search to narrow down results further
- Add [[https://github.com/khoj-ai/khoj/#query-filters][query filters]] during search to narrow down results further
e.g `What is the meaning of life? -"god" +"none" dt>"last week"`

880
src/interface/emacs/khoj.el
View File

File diff suppressed because it is too large Load Diff

212
src/interface/emacs/tests/khoj-tests.el Normal file
View File

@@ -0,0 +1,212 @@
;;; khoj-tests.el --- Test suite for khoj.el -*- lexical-binding: t -*-
;; Copyright (C) 2023 Debanjum Singh Solanky
;; Author: Debanjum Singh Solanky <debanjum@gmail.com>
;; Version: 0.0.0
;; Package-Requires: ((emacs "27.1") (transient "0.3.0") (dash "2.19.1") (org "9.0.0"))
;; URL: https://github.com/khoj-ai/khoj/tree/master/src/interface/emacs
;;; License:
;; This program is free software; you can redistribute it and/or
;; modify it under the terms of the GNU General Public License
;; as published by the Free Software Foundation; either version 3
;; of the License, or (at your option) any later version.
;; This program is distributed in the hope that it will be useful,
;; but WITHOUT ANY WARRANTY; without even the implied warranty of
;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
;; GNU General Public License for more details.
;; You should have received a copy of the GNU General Public License
;; along with this program. If not, see <http://www.gnu.org/licenses/>.
;;; Commentary:
;; This file contains the test suite for khoj.el.
;;; Code:
(require 'dash)
(require 'ert)
(require 'khoj)
(require 'org)
;; ----------------------------------------------------
;; Test Extract and Render Entries of each Content Type
;; ----------------------------------------------------
(ert-deftest khoj-tests--extract-entries-as-markdown ()
"Test `json-response', `query' from API formatted as markdown."
(let ((user-query "Become God")
(json-response-from-khoj-backend
(json-read-from-string
"[\
{\
\"entry\": \"## Upgrade\\n\\n Penance to Immortality\",\
\"score\": \"0.376\",\
\"additional\": {\
\"file\": \"/home/ravan/upgrade.md\",\
\"compiled\": \"## Upgrade Penance to Immortality\"\
}\
},\
{\
\"entry\": \"## Act\\n\\n Rule everything\",\
\"score\": \"0.153\",\
\"additional\": {\
\"file\": \"/home/ravan/act.md\",\
\"compiled\": \"## Act Rule everything\"\
}\
}]\
")))
(should
(equal
(khoj--extract-entries-as-markdown json-response-from-khoj-backend user-query)
"\
# Become God\n\
## Upgrade\n\
\n\
Penance to Immortality\n\n\
## Act\n\
\n\
Rule everything\n\n"))))
(ert-deftest khoj-tests--extract-entries-as-org ()
"Test `json-response', `query' from API formatted as org."
(let ((user-query "Become God")
(json-response-from-khoj-backend
(json-read-from-string
"[\
{\
\"entry\": \"** Upgrade\\n\\n Penance to Immortality\\n\",\
\"score\": \"0.42\",\
\"additional\": {\
\"file\": \"/home/ravan/upgrade.md\",\
\"compiled\": \"** Upgrade Penance to Immortality\"\
}\
},\
{\
\"entry\": \"** Act\\n\\n Rule everything\\n\",\
\"score\": \"0.42\",\
\"additional\": {\
\"file\": \"/home/ravan/act.md\",\
\"compiled\": \"** Act Rule everything\"\
}\
}]\
")))
(should
(equal
(khoj--extract-entries-as-org json-response-from-khoj-backend user-query)
"\
* Become God\n\
** Upgrade\n\
\n\
Penance to Immortality\n\
** Act\n\
\n\
Rule everything\n\
\n"))))
;; -------------------------------------
;; Test Helpers for Find Similar Feature
;; -------------------------------------
(ert-deftest khoj-tests--get-current-outline-entry-text ()
"Test get current outline-mode entry text'."
(with-temp-buffer
(insert "\
* Become God\n\
** Upgrade\n\
\n\
Penance to Immortality\n\
** Act\n\
\n\
Rule everything\\n")
(goto-char (point-min))
;; Test getting current entry text from cursor at start of outline heading
(outline-next-visible-heading 1)
(should
(equal
(khoj--get-current-outline-entry-text)
"\
** Upgrade\n\
\n\
Penance to Immortality"))
;; Test getting current entry text from cursor within outline entry
(forward-line)
(should
(equal
(khoj--get-current-outline-entry-text)
"\
** Upgrade\n\
\n\
Penance to Immortality"))
))
(ert-deftest khoj-tests--get-current-paragraph-text ()
"Test get current paragraph text'."
(with-temp-buffer
(insert "\
* Become God\n\
** Upgrade\n\
\n\
Penance to Immortality\n\
** Act\n\
\n\
Rule everything\n")
;; Test getting current paragraph text from cursor at start of buffer
(goto-char (point-min))
(should
(equal
(khoj--get-current-paragraph-text)
"* Become God\n\
** Upgrade"))
;; Test getting current paragraph text from cursor within paragraph
(goto-char (point-min))
(forward-line 1)
(should
(equal
(khoj--get-current-paragraph-text)
"* Become God\n\
** Upgrade"))
;; Test getting current paragraph text from cursor at paragraph end
(goto-char (point-min))
(forward-line 2)
(should
(equal
(khoj--get-current-paragraph-text)
"* Become God\n\
** Upgrade"))
;; Test getting current paragraph text from cursor at start of middle paragraph
(goto-char (point-min))
(forward-line 3)
(should
(equal
(khoj--get-current-paragraph-text)
"Penance to Immortality\n\
** Act"))
;; Test getting current paragraph text from cursor at end of buffer
(goto-char (point-max))
(should
(equal
(khoj--get-current-paragraph-text)
"Rule everything"))
))
(provide 'khoj-tests)
;;; khoj-tests.el ends here

2
src/interface/obsidian/.eslintignore
View File

@@ -1,2 +1,2 @@
npm node_modules
build
build

6
src/interface/obsidian/.eslintrc
View File

@@ -9,7 +9,7 @@
"eslint:recommended",
"plugin:@typescript-eslint/eslint-recommended",
"plugin:@typescript-eslint/recommended"
],
],
"parserOptions": {
"sourceType": "module"
},
@@ -19,5 +19,5 @@
"@typescript-eslint/ban-ts-comment": "off",
"no-prototype-builtins": "off",
"@typescript-eslint/no-empty-function": "off"
}
}
}
}

2
src/interface/obsidian/.npmrc
View File

@@ -1 +1 @@
tag-version-prefix=""
tag-version-prefix=""

1
src/interface/obsidian/LICENSE
View File

@@ -619,4 +619,3 @@ Program, unless a warranty or assumption of liability accompanies a
copy of the Program in return for a fee.
END OF TERMS AND CONDITIONS

93
src/interface/obsidian/README.md
View File

@@ -1,17 +1,21 @@
# Khoj Obsidian 🦅
> Natural language search for your Obsidian notes using [Khoj](https://github.com/debanjum/khoj)
<img src="/src/khoj/interface/web/assets/icons/khoj-logo-sideways.svg" width="200" alt="Khoj Logo">Obsidian
> An AI personal assistant for your Digital Brain in Obsidian
## Table of Contents
- [Features](#Features)
- [Demo](#Demo)
- [Description](#Description)
- [Interface](#Interface)
- [Search Demo](#Search-Demo)
- [Interfaces](#Interfaces)
- [Search Modal](#Search-Modal)
- [Chat Modal](#Chat-Modal)
- [Setup](#Setup)
- [Setup Backend](#1-Setup-Backend)
- [Setup Plugin](#2-Setup-Plugin)
- [Use](#Use)
- [Search](#search)
- [Chat](#chat)
- [Find Similar Notes](#find-similar-notes)
- [Upgrade](#Upgrade)
- [Upgrade Backend](#1-Upgrade-Backend)
@@ -21,45 +25,84 @@
- [Implementation](#Implementation)
## Features
- **Natural**: Advanced natural language understanding using Transformer based ML Models
- **Local**: Your personal data stays local. All search, indexing is done on your machine[\*](https://github.com/debanjum/khoj#miscellaneous)
- **Incremental**: Incremental search for a fast, search-as-you-type experience
- **Search**
- **Natural**: Advanced natural language understanding using Transformer based ML Models
- **Local**: Your personal data stays local. All search and indexing is done on your machine. *Unlike chat which requires access to GPT.*
- **Incremental**: Incremental search for a fast, search-as-you-type experience
- **Chat**
- **Faster answers**: Find answers faster and with less effort than search
- **Iterative discovery**: Iteratively explore and (re-)discover your notes
- **Assisted creativity**: Smoothly weave across answers retrieval and content generation
## Demo
https://user-images.githubusercontent.com/6413477/210486007-36ee3407-e6aa-4185-8a26-b0bfc0a4344f.mp4
### Search Demo
https://github.com/khoj-ai/khoj/assets/6413477/3e33d8ea-25bb-46c8-a3bf-c92f78d0f56b
<details><summary>Description</summary>
1. Install Khoj via `pip` and start Khoj backend in non-gui mode
1. Install Khoj via `pip` and start Khoj backend
```shell
python -m pip install khoj-assistant && khoj
```
2. Install Khoj plugin via Community Plugins settings pane on Obsidian app
3. Check the new Khoj plugin settings
4. Wait for Khoj backend to index markdown files in the current Vault
5. Open Khoj plugin on Obsidian via Search button on Left Pane
6. Search \"*Announce plugin to folks*\" in the [Obsidian Plugin docs](https://marcus.se.net/obsidian-plugin-docs/)
7. Jump to the [search result](https://marcus.se.net/obsidian-plugin-docs/publishing/submit-your-plugin)
- Check the new Khoj plugin settings
- Wait for Khoj backend to index markdown, PDF files in the current Vault
- Open Khoj plugin on Obsidian via Search button on Left Pane
- Search \"*Announce plugin to folks*\" in the [Obsidian Plugin docs](https://marcus.se.net/obsidian-plugin-docs/)
- Jump to the [search result](https://marcus.se.net/obsidian-plugin-docs/publishing/submit-your-plugin)
</details>
### Interface
![](https://github.com/debanjum/khoj/blob/master/src/interface/obsidian/docs/khoj_on_obsidian_0.2.5.png?)
## Interfaces
### Search Modal
![](https://github.com/khoj-ai/khoj/blob/master/src/interface/obsidian/docs/khoj_on_obsidian_0.2.5.png?)
### Chat Modal
![](https://github.com/khoj-ai/khoj/blob/master/src/interface/obsidian/docs/khoj_chat_on_obsidian_0.6.0.png?)
## Setup
- *Make sure [python](https://realpython.com/installing-python/) and [pip](https://pip.pypa.io/en/stable/installation/) are installed on your machine*
- *Ensure you follow the ordering of the setup steps. Install the plugin after starting the khoj backend. This allows the plugin to configure the khoj backend*
### 1. Setup Backend
Open terminal/cmd and run below command to install and start the khoj backend
- On Linux/MacOS
```shell
python -m pip install khoj-assistant && khoj
```
- On Windows
```shell
py -m pip install khoj-assistant && khoj
```
```shell
pip install khoj-assistant && khoj --no-gui
```
### 2. Setup Plugin
1. Open *Community plugins* tab in Obsidian settings panel
2. Click Browse and Search for *Khoj*
3. Click *Install*, after that click *Enable* on the Khoj plugin
1. Open [Khoj](https://obsidian.md/plugins?id=khoj) from the *Community plugins* tab in Obsidian settings panel
2. Click *Install*, then *Enable* on the Khoj plugin page in Obsidian
3. [Optional] To enable Khoj Chat, set your [OpenAI API key](https://platform.openai.com/account/api-keys) in the Khoj plugin settings
See [official docs](https://help.obsidian.md/Advanced+topics/Community+plugins#Discover+and+install+community+plugins) for details
See [official Obsidian plugin docs](https://help.obsidian.md/Extending+Obsidian/Community+plugins) for details
## Use
### Chat
Run *Khoj: Chat* from the [Command Palette](https://help.obsidian.md/Plugins/Command+palette) and ask questions in a natural, conversational style.<br />
E.g "When did I file my taxes last year?"
Notes:
- *Using Khoj Chat will result in query relevant notes being shared with OpenAI for ChatGPT to respond.*
- *To use Khoj Chat, ensure you've set your [OpenAI API key](https://platform.openai.com/account/api-keys) in the Khoj plugin settings.*
See [Khoj Chat](https://github.com/khoj-ai/khoj/tree/master/#Khoj-Chat) for more details
![](https://github.com/khoj-ai/khoj/blob/master/src/interface/obsidian/docs/khoj_chat_on_obsidian_0.6.0.png?)
### Search
Click the *Khoj search* icon 🔎 on the [Ribbon](https://help.obsidian.md/User+interface/Workspace/Ribbon) or run *Khoj: Search* from the [Command Palette](https://help.obsidian.md/Plugins/Command+palette)
*Note: Ensure the khoj server is running in the background before searching. Execute `khoj` in your terminal if it is not already running*
https://user-images.githubusercontent.com/6413477/218801155-cd67e8b4-a770-404a-8179-d6b61caa0f93.mp4
<details><summary>Query Filters</summary>
Use structured query syntax to filter the natural language search results
@@ -105,14 +148,14 @@ To see other notes similar to the current one, run *Khoj: Find Similar Notes* fr
So notes across multiple vaults **cannot** be searched at the same time
## Visualize Codebase
<img src="https://github.com/debanjum/khoj/blob/master/src/interface/obsidian/docs/khoj_obsidian_codebase_visualization_0.2.1.png" width="700" />
<img src="https://github.com/khoj-ai/khoj/blob/master/src/interface/obsidian/docs/khoj_obsidian_codebase_visualization_0.2.1.png" width="700" />
## Implementation
The plugin implements the following functionality to search your notes with Khoj:
- [X] Open the Khoj search modal via left ribbon icon or the *Khoj: Search* command
- [X] Render results as Markdown preview to improve readability
- [X] Configure Khoj via the plugin setting tab on the settings page
- Set Obsidian Vault to Index with Khoj. Defaults to all markdown files in current Vault
- Set Obsidian Vault to Index with Khoj. Defaults to all markdown, PDF files in current Vault
- Set URL of Khoj backend
- Set Number of Search Results to show in Search Modal
- [X] Allow reranking of result to improve search quality

BIN
src/interface/obsidian/docs/khoj_chat_on_obsidian_0.6.0.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 277 KiB

10
src/interface/obsidian/esbuild.config.mjs
View File

@@ -31,6 +31,16 @@ esbuild.build({
'@lezer/common',
'@lezer/highlight',
'@lezer/lr',
'node:fs',
'node:path',
'node:util',
'node:url',
'node:http',
'node:https',
'node:stream',
'node:zlib',
'node:buffer',
'node:net',
...builtins],
format: 'cjs',
watch: !prod,

4
src/interface/obsidian/manifest.json
View File

@@ -1,9 +1,9 @@
{
"id": "khoj",
"name": "Khoj",
"version": "0.2.5",
"version": "0.8.0",
"minAppVersion": "0.15.0",
"description": "Natural, Incremental Search for your Second Brain 🦅",
"description": "An AI Personal Assistant for your Digital Brain",
"author": "Debanjum Singh Solanky",
"authorUrl": "https://github.com/debanjum",
"isDesktopOnly": false

17
src/interface/obsidian/package.json
View File

@@ -1,16 +1,21 @@
{
"name": "Khoj",
"version": "0.2.5",
"description": "Natural, Incremental Search for your Second Brain 🦅",
"version": "0.8.0",
"description": "An AI Personal Assistant for your Digital Brain",
"main": "src/main.js",
"scripts": {
"dev": "node esbuild.config.mjs",
"build": "tsc -noEmit -skipLibCheck && node esbuild.config.mjs production",
"version": "node version-bump.mjs && git add manifest.json versions.json"
},
"keywords": ["search"],
"keywords": [
"search",
"chat",
"AI",
"assistant"
],
"author": "Debanjum Singh Solanky",
"license": "GPLv3",
"license": "GPL-3.0-or-later",
"devDependencies": {
"@types/node": "^16.11.6",
"@typescript-eslint/eslint-plugin": "5.29.0",
@@ -20,5 +25,9 @@
"obsidian": "latest",
"tslib": "2.4.0",
"typescript": "4.7.4"
},
"dependencies": {
"@types/node-fetch": "^2.6.4",
"node-fetch": "^3.1.0"
}
}

194
src/interface/obsidian/src/chat_modal.ts Normal file
View File

@@ -0,0 +1,194 @@
import { App, Modal, request } from 'obsidian';
import { KhojSetting } from 'src/settings';
import fetch from "node-fetch";
export class KhojChatModal extends Modal {
result: string;
setting: KhojSetting;
constructor(app: App, setting: KhojSetting) {
super(app);
this.setting = setting;
// Register Modal Keybindings to send user message
this.scope.register([], 'Enter', async () => {
// Get text in chat input elmenet
let input_el = <HTMLInputElement>this.contentEl.getElementsByClassName("khoj-chat-input")[0];
// Clear text after extracting message to send
let user_message = input_el.value;
input_el.value = "";
// Get and render chat response to user message
await this.getChatResponse(user_message);
});
}
async onOpen() {
let { contentEl } = this;
contentEl.addClass("khoj-chat");
// Add title to the Khoj Chat modal
contentEl.createEl("h1", ({ attr: { id: "khoj-chat-title" }, text: "Khoj Chat" }));
// Create area for chat logs
contentEl.createDiv({ attr: { id: "khoj-chat-body", class: "khoj-chat-body" } });
// Get chat history from Khoj backend
await this.getChatHistory();
// Add chat input field
contentEl.createEl("input",
{
attr: {
type: "text",
id: "khoj-chat-input",
autofocus: "autofocus",
placeholder: "Chat with Khoj [Hit Enter to send message]",
class: "khoj-chat-input option"
}
})
.addEventListener('change', (event) => { this.result = (<HTMLInputElement>event.target).value });
// Scroll to bottom of modal, till the send message input box
this.modalEl.scrollTop = this.modalEl.scrollHeight;
}
generateReference(messageEl: any, reference: string, index: number) {
// Generate HTML for Chat Reference
// `<sup><abbr title="${escaped_ref}" tabindex="0">${index}</abbr></sup>`;
let escaped_ref = reference.replace(/"/g, "&quot;")
return messageEl.createEl("sup").createEl("abbr", {
attr: {
title: escaped_ref,
tabindex: "0",
},
text: `[${index}] `,
});
}
renderMessageWithReferences(message: string, sender: string, context?: [string], dt?: Date) {
let messageEl = this.renderMessage(message, sender, dt);
if (context && !!messageEl) {
context.map((reference, index) => this.generateReference(messageEl, reference, index + 1));
}
}
renderMessage(message: string, sender: string, dt?: Date): Element | null {
let message_time = this.formatDate(dt ?? new Date());
let emojified_sender = sender == "khoj" ? "🏮 Khoj" : "🤔 You";
// Append message to conversation history HTML element.
// The chat logs should display above the message input box to follow standard UI semantics
let chat_body_el = this.contentEl.getElementsByClassName("khoj-chat-body")[0];
let chat_message_el = chat_body_el.createDiv({
attr: {
"data-meta": `${emojified_sender} at ${message_time}`,
class: `khoj-chat-message ${sender}`
},
}).createDiv({
attr: {
class: `khoj-chat-message-text ${sender}`
},
text: `${message}`
})
// Remove user-select: none property to make text selectable
chat_message_el.style.userSelect = "text";
// Scroll to bottom after inserting chat messages
this.modalEl.scrollTop = this.modalEl.scrollHeight;
return chat_message_el
}
createKhojResponseDiv(dt?: Date): HTMLDivElement {
let message_time = this.formatDate(dt ?? new Date());
// Append message to conversation history HTML element.
// The chat logs should display above the message input box to follow standard UI semantics
let chat_body_el = this.contentEl.getElementsByClassName("khoj-chat-body")[0];
let chat_message_el = chat_body_el.createDiv({
attr: {
"data-meta": `🏮 Khoj at ${message_time}`,
class: `khoj-chat-message khoj`
},
}).createDiv({
attr: {
class: `khoj-chat-message-text khoj`
},
})
// Scroll to bottom after inserting chat messages
this.modalEl.scrollTop = this.modalEl.scrollHeight;
return chat_message_el
}
renderIncrementalMessage(htmlElement: HTMLDivElement, additionalMessage: string) {
htmlElement.innerHTML += additionalMessage;
// Scroll to bottom of modal, till the send message input box
this.modalEl.scrollTop = this.modalEl.scrollHeight;
}
formatDate(date: Date): string {
// Format date in HH:MM, DD MMM YYYY format
let time_string = date.toLocaleTimeString('en-IN', { hour: '2-digit', minute: '2-digit', hour12: false });
let date_string = date.toLocaleString('en-IN', { year: 'numeric', month: 'short', day: '2-digit' }).replace(/-/g, ' ');
return `${time_string}, ${date_string}`;
}
async getChatHistory(): Promise<void> {
// Get chat history from Khoj backend
let chatUrl = `${this.setting.khojUrl}/api/chat/history?client=obsidian`;
let response = await request(chatUrl);
let chatLogs = JSON.parse(response).response;
chatLogs.forEach((chatLog: any) => {
this.renderMessageWithReferences(chatLog.message, chatLog.by, chatLog.context, new Date(chatLog.created));
});
}
async getChatResponse(query: string | undefined | null): Promise<void> {
// Exit if query is empty
if (!query || query === "") return;
// Render user query as chat message
this.renderMessage(query, "you");
// Get chat response from Khoj backend
let encodedQuery = encodeURIComponent(query);
let chatUrl = `${this.setting.khojUrl}/api/chat?q=${encodedQuery}&n=${this.setting.resultsCount}&client=obsidian&stream=true`;
let responseElement = this.createKhojResponseDiv();
// Temporary status message to indicate that Khoj is thinking
this.renderIncrementalMessage(responseElement, "🤔");
let response = await fetch(chatUrl, {
method: "GET",
headers: {
"Access-Control-Allow-Origin": "*",
"Content-Type": "text/event-stream"
},
})
try {
if (response.body == null) {
throw new Error("Response body is null");
}
// Clear thinking status message
if (responseElement.innerHTML === "🤔") {
responseElement.innerHTML = "";
}
for await (const chunk of response.body) {
const responseText = chunk.toString();
if (responseText.startsWith("### compiled references:")) {
return;
}
this.renderIncrementalMessage(responseElement, responseText);
}
} catch (err) {
this.renderIncrementalMessage(responseElement, "Sorry, unable to get response from Khoj backend ❤️‍🩹. Contact developer for help at team@khoj.dev or <a href='https://discord.gg/BDgyabRM6e'>in Discord</a>")
}
}
}

34
src/interface/obsidian/src/main.ts
View File

@@ -1,6 +1,7 @@
import { Notice, Plugin } from 'obsidian';
import { KhojSetting, KhojSettingTab, DEFAULT_SETTINGS } from 'src/settings'
import { KhojModal } from 'src/modal'
import { KhojSearchModal } from 'src/search_modal'
import { KhojChatModal } from 'src/chat_modal'
import { configureKhojBackend } from './utils';
@@ -16,7 +17,7 @@ export default class Khoj extends Plugin {
name: 'Search',
checkCallback: (checking) => {
if (!checking && this.settings.connectedToBackend)
new KhojModal(this.app, this.settings).open();
new KhojSearchModal(this.app, this.settings).open();
return this.settings.connectedToBackend;
}
});
@@ -24,19 +25,30 @@ export default class Khoj extends Plugin {
// Add similar notes command. It can only be triggered from the editor
this.addCommand({
id: 'similar',
name: 'Find Similar Notes',
name: 'Find similar notes',
editorCheckCallback: (checking) => {
if (!checking && this.settings.connectedToBackend)
new KhojModal(this.app, this.settings, true).open();
new KhojSearchModal(this.app, this.settings, true).open();
return this.settings.connectedToBackend;
}
});
// Add chat command. It can be triggered from anywhere
this.addCommand({
id: 'chat',
name: 'Chat',
checkCallback: (checking) => {
if (!checking && this.settings.connectedToBackend && !!this.settings.openaiApiKey)
new KhojChatModal(this.app, this.settings).open();
return !!this.settings.openaiApiKey;
}
});
// Create an icon in the left ribbon.
this.addRibbonIcon('search', 'Khoj', (_: MouseEvent) => {
// Called when the user clicks the icon.
this.settings.connectedToBackend
? new KhojModal(this.app, this.settings).open()
? new KhojSearchModal(this.app, this.settings).open()
: new Notice(`❗️Ensure Khoj backend is running and Khoj URL is pointing to it in the plugin settings`);
});
@@ -48,12 +60,16 @@ export default class Khoj extends Plugin {
// Load khoj obsidian plugin settings
this.settings = Object.assign({}, DEFAULT_SETTINGS, await this.loadData());
// Load, configure khoj server settings
await configureKhojBackend(this.settings);
if (this.settings.autoConfigure) {
// Load, configure khoj server settings
await configureKhojBackend(this.app.vault, this.settings);
}
}
async saveSettings() {
await configureKhojBackend(this.settings, false)
.then(() => this.saveData(this.settings));
if (this.settings.autoConfigure) {
await configureKhojBackend(this.app.vault, this.settings, false);
}
this.saveData(this.settings);
}
}

118
src/interface/obsidian/src/modal.ts
View File

@@ -1,118 +0,0 @@
import { App, SuggestModal, request, MarkdownRenderer, Instruction, Platform } from 'obsidian';
import { KhojSetting } from 'src/settings';
export interface SearchResult {
entry: string;
file: string;
}
export class KhojModal extends SuggestModal<SearchResult> {
setting: KhojSetting;
rerank: boolean = false;
find_similar_notes: boolean;
constructor(app: App, setting: KhojSetting, find_similar_notes: boolean = false) {
super(app);
this.setting = setting;
this.find_similar_notes = find_similar_notes;
// Hide input element in Similar Notes mode
this.inputEl.hidden = this.find_similar_notes;
// Register Modal Keybindings to Rerank Results
this.scope.register(['Mod'], 'Enter', async () => {
// Re-rank when explicitly triggered by user
this.rerank = true
// Trigger input event to get and render (reranked) results from khoj backend
this.inputEl.dispatchEvent(new Event('input'));
// Rerank disabled by default to satisfy latency requirements for incremental search
this.rerank = false
});
// Add Hints to Modal for available Keybindings
const modalInstructions: Instruction[] = [
{
command: '↑↓',
purpose: 'to navigate',
},
{
command: '↵',
purpose: 'to open',
},
{
command: Platform.isMacOS ? 'cmd ↵' : 'ctrl ↵',
purpose: 'to rerank',
},
{
command: 'esc',
purpose: 'to dismiss',
},
]
this.setInstructions(modalInstructions);
// Set Placeholder Text for Modal
this.setPlaceholder('Search with Khoj 🦅...');
}
async onOpen() {
if (this.find_similar_notes) {
// If markdown file is currently active
let file = this.app.workspace.getActiveFile();
if (file && file.extension === 'md') {
// Enable rerank of search results
this.rerank = true
// Set contents of active markdown file to input element
this.inputEl.value = await this.app.vault.read(file);
// Trigger search to get and render similar notes from khoj backend
this.inputEl.dispatchEvent(new Event('input'));
this.rerank = false
}
else {
this.resultContainerEl.setText('Cannot find similar notes for non-markdown files');
}
}
}
async getSuggestions(query: string): Promise<SearchResult[]> {
// Query Khoj backend for search results
let encodedQuery = encodeURIComponent(query);
let searchUrl = `${this.setting.khojUrl}/api/search?q=${encodedQuery}&n=${this.setting.resultsCount}&r=${this.rerank}&t=markdown`
let results = await request(searchUrl)
.then(response => JSON.parse(response))
.then(data => data
.filter((result: any) => !this.find_similar_notes || !result.additional.file.endsWith(this.app.workspace.getActiveFile()?.path))
.map((result: any) => { return { entry: result.entry, file: result.additional.file } as SearchResult; }));
return results;
}
async renderSuggestion(result: SearchResult, el: HTMLElement) {
let words_to_render = 30;
let entry_words = result.entry.split(' ')
let entry_snipped_indicator = entry_words.length > words_to_render ? ' **...**' : '';
let snipped_entry = entry_words.slice(0, words_to_render).join(' ');
MarkdownRenderer.renderMarkdown(snipped_entry + entry_snipped_indicator, el, null, null);
}
async onChooseSuggestion(result: SearchResult, _: MouseEvent | KeyboardEvent) {
// Get all markdown files in vault
const mdFiles = this.app.vault.getMarkdownFiles();
// Find the vault file matching file of result. Open file at result heading
mdFiles
// Sort by descending length of path
// This finds longest path match when multiple files have same name
.sort((a, b) => b.path.length - a.path.length)
.forEach((file) => {
// Find best file match across operating systems
// E.g Khoj Server on Linux, Obsidian Vault on Android
if (result.file.endsWith(file.path)) {
let resultHeading = result.entry.split('\n', 1)[0];
let linkToEntry = `${file.path}${resultHeading}`
this.app.workspace.openLinkText(linkToEntry, '');
console.log(`Link: ${linkToEntry}, File: ${file.path}, Heading: ${resultHeading}`);
return
}
});
}
}

169
src/interface/obsidian/src/search_modal.ts Normal file
View File

@@ -0,0 +1,169 @@
import { App, SuggestModal, request, MarkdownRenderer, Instruction, Platform } from 'obsidian';
import { KhojSetting } from 'src/settings';
import { createNoteAndCloseModal } from 'src/utils';
export interface SearchResult {
entry: string;
file: string;
}
export class KhojSearchModal extends SuggestModal<SearchResult> {
setting: KhojSetting;
rerank: boolean = false;
find_similar_notes: boolean;
query: string = "";
app: App;
constructor(app: App, setting: KhojSetting, find_similar_notes: boolean = false) {
super(app);
this.app = app;
this.setting = setting;
this.find_similar_notes = find_similar_notes;
// Hide input element in Similar Notes mode
this.inputEl.hidden = this.find_similar_notes;
// Register Modal Keybindings to Rerank Results
this.scope.register(['Mod'], 'Enter', async () => {
// Re-rank when explicitly triggered by user
this.rerank = true
// Trigger input event to get and render (reranked) results from khoj backend
this.inputEl.dispatchEvent(new Event('input'));
// Rerank disabled by default to satisfy latency requirements for incremental search
this.rerank = false
});
// Register Modal Keybindings to Create New Note with Query as Title
this.scope.register(['Shift'], 'Enter', async () => {
if (this.query != "") createNoteAndCloseModal(this.query, this);
});
this.scope.register(['Ctrl', 'Shift'], 'Enter', async () => {
if (this.query != "") createNoteAndCloseModal(this.query, this, { newLeaf: true });
});
// Add Hints to Modal for available Keybindings
const modalInstructions: Instruction[] = [
{
command: '↑↓',
purpose: 'to navigate',
},
{
command: '↵',
purpose: 'to open',
},
{
command: Platform.isMacOS ? 'cmd ↵' : 'ctrl ↵',
purpose: 'to rerank',
},
{
command: 'esc',
purpose: 'to dismiss',
},
]
this.setInstructions(modalInstructions);
// Set Placeholder Text for Modal
this.setPlaceholder('Search with Khoj...');
}
async onOpen() {
if (this.find_similar_notes) {
// If markdown file is currently active
let file = this.app.workspace.getActiveFile();
if (file && file.extension === 'md') {
// Enable rerank of search results
this.rerank = true
// Set input element to contents of active markdown file
// truncate to first 8,000 characters to avoid hitting query size limits
this.inputEl.value = await this.app.vault.read(file).then(file_str => file_str.slice(0, 8000));
// Trigger search to get and render similar notes from khoj backend
this.inputEl.dispatchEvent(new Event('input'));
this.rerank = false
}
else {
this.resultContainerEl.setText('Cannot find similar notes for non-markdown files');
}
}
}
async getSuggestions(query: string): Promise<SearchResult[]> {
// Query Khoj backend for search results
let encodedQuery = encodeURIComponent(query);
let searchUrl = `${this.setting.khojUrl}/api/search?q=${encodedQuery}&n=${this.setting.resultsCount}&r=${this.rerank}&client=obsidian`;
// Get search results for markdown and pdf files
let mdResponse = await request(`${searchUrl}&t=markdown`);
let pdfResponse = await request(`${searchUrl}&t=pdf`);
// Parse search results
let mdData = JSON.parse(mdResponse)
.filter((result: any) => !this.find_similar_notes || !result.additional.file.endsWith(this.app.workspace.getActiveFile()?.path))
.map((result: any) => { return { entry: result.entry, score: result.score, file: result.additional.file }; });
let pdfData = JSON.parse(pdfResponse)
.filter((result: any) => !this.find_similar_notes || !result.additional.file.endsWith(this.app.workspace.getActiveFile()?.path))
.map((result: any) => { return { entry: `## ${result.additional.compiled}`, score: result.score, file: result.additional.file } as SearchResult; })
// Combine markdown and PDF results and sort them by score
let results = mdData.concat(pdfData)
.sort((a: any, b: any) => b.score - a.score)
.map((result: any) => { return { entry: result.entry, file: result.file } as SearchResult; })
this.query = query;
return results;
}
async renderSuggestion(result: SearchResult, el: HTMLElement) {
// Max number of lines to render
let lines_to_render = 8;
// Extract filename of result
let os_path_separator = result.file.includes('\\') ? '\\' : '/';
let filename = result.file.split(os_path_separator).pop();
// Remove YAML frontmatter when rendering string
result.entry = result.entry.replace(/---[\n\r][\s\S]*---[\n\r]/, '');
// Truncate search results to lines_to_render
let entry_snipped_indicator = result.entry.split('\n').length > lines_to_render ? ' **...**' : '';
let snipped_entry = result.entry.split('\n').slice(0, lines_to_render).join('\n');
// Show reindex hint on first search result
if (this.resultContainerEl.children.length == 1) {
let infoHintEl = createEl("div",{ cls: 'khoj-info-hint' });
el.insertAdjacentElement("beforebegin", infoHintEl);
setTimeout(() => {
infoHintEl.setText('Unexpected results? Try re-index your vault from the Khoj plugin settings to fix it.');
}, 3000);
}
// Show filename of each search result for context
el.createEl("div",{ cls: 'khoj-result-file' }).setText(filename ?? "");
let result_el = el.createEl("div", { cls: 'khoj-result-entry' })
// @ts-ignore
MarkdownRenderer.renderMarkdown(snipped_entry + entry_snipped_indicator, result_el, null, null);
}
async onChooseSuggestion(result: SearchResult, _: MouseEvent | KeyboardEvent) {
// Get all markdown and PDF files in vault
const mdFiles = this.app.vault.getMarkdownFiles();
const pdfFiles = this.app.vault.getFiles().filter(file => file.extension === 'pdf');
// Find the vault file matching file of chosen search result
let file_match = mdFiles.concat(pdfFiles)
// Sort by descending length of path
// This finds longest path match when multiple files have same name
.sort((a, b) => b.path.length - a.path.length)
// The first match is the best file match across OS
// e.g Khoj server on Linux, Obsidian vault on Android
.find(file => result.file.replace(/\\/g, "/").endsWith(file.path))
// Open vault file at heading of chosen search result
if (file_match) {
let resultHeading = file_match.extension !== 'pdf' ? result.entry.split('\n', 1)[0] : '';
let linkToEntry = resultHeading.startsWith('#') ? `${file_match.path}${resultHeading}` : file_match.path;
this.app.workspace.openLinkText(linkToEntry, '');
console.log(`Link: ${linkToEntry}, File: ${file_match.path}, Heading: ${resultHeading}`);
}
}
}

72
src/interface/obsidian/src/settings.ts
View File

@@ -2,15 +2,19 @@ import { App, Notice, PluginSettingTab, request, Setting } from 'obsidian';
import Khoj from 'src/main';
export interface KhojSetting {
openaiApiKey: string;
resultsCount: number;
khojUrl: string;
connectedToBackend: boolean;
autoConfigure: boolean;
}
export const DEFAULT_SETTINGS: KhojSetting = {
resultsCount: 6,
khojUrl: 'http://localhost:8000',
khojUrl: 'http://127.0.0.1:8000',
connectedToBackend: false,
autoConfigure: true,
openaiApiKey: '',
}
export class KhojSettingTab extends PluginSettingTab {
@@ -36,12 +40,21 @@ export class KhojSettingTab extends PluginSettingTab {
.setValue(`${this.plugin.settings.khojUrl}`)
.onChange(async (value) => {
this.plugin.settings.khojUrl = value.trim();
await this.plugin.saveSettings()
.finally(() => containerEl.firstElementChild?.setText(this.getBackendStatusMessage()));
await this.plugin.saveSettings();
containerEl.firstElementChild?.setText(this.getBackendStatusMessage());
}));
new Setting(containerEl)
new Setting(containerEl)
.setName('OpenAI API Key')
.setDesc('Your OpenAI API Key for Khoj Chat')
.addText(text => text
.setValue(`${this.plugin.settings.openaiApiKey}`)
.onChange(async (value) => {
this.plugin.settings.openaiApiKey = value.trim();
await this.plugin.saveSettings();
}));
new Setting(containerEl)
.setName('Results Count')
.setDesc('The number of search results to show')
.setDesc('The number of results to show in search and use for chat')
.addSlider(slider => slider
.setLimits(1, 10, 1)
.setValue(this.plugin.settings.resultsCount)
@@ -50,6 +63,15 @@ export class KhojSettingTab extends PluginSettingTab {
this.plugin.settings.resultsCount = value;
await this.plugin.saveSettings();
}));
new Setting(containerEl)
.setName('Auto Configure')
.setDesc('Automatically configure the Khoj backend')
.addToggle(toggle => toggle
.setValue(this.plugin.settings.autoConfigure)
.onChange(async (value) => {
this.plugin.settings.autoConfigure = value;
await this.plugin.saveSettings();
}));
let indexVaultSetting = new Setting(containerEl);
indexVaultSetting
.setName('Index Vault')
@@ -59,16 +81,40 @@ export class KhojSettingTab extends PluginSettingTab {
.setCta()
.onClick(async () => {
// Disable button while updating index
button.setButtonText('Updating...');
button.removeCta()
button.setButtonText('Updating 🌑');
button.removeCta();
indexVaultSetting = indexVaultSetting.setDisabled(true);
await request(`${this.plugin.settings.khojUrl}/api/update?t=markdown&force=true`)
.then(() => new Notice('✅ Updated Khoj index.'));
// Show indicator for indexing in progress
const progress_indicator = window.setInterval(() => {
if (button.buttonEl.innerText === 'Updating 🌑') {
button.setButtonText('Updating 🌘');
} else if (button.buttonEl.innerText === 'Updating 🌘') {
button.setButtonText('Updating 🌗');
} else if (button.buttonEl.innerText === 'Updating 🌗') {
button.setButtonText('Updating 🌖');
} else if (button.buttonEl.innerText === 'Updating 🌖') {
button.setButtonText('Updating 🌕');
} else if (button.buttonEl.innerText === 'Updating 🌕') {
button.setButtonText('Updating 🌔');
} else if (button.buttonEl.innerText === 'Updating 🌔') {
button.setButtonText('Updating 🌓');
} else if (button.buttonEl.innerText === 'Updating 🌓') {
button.setButtonText('Updating 🌒');
} else if (button.buttonEl.innerText === 'Updating 🌒') {
button.setButtonText('Updating 🌑');
}
}, 300);
this.plugin.registerInterval(progress_indicator);
// Re-enable button once index is updated
await request(`${this.plugin.settings.khojUrl}/api/update?t=markdown&force=true&client=obsidian`);
await request(`${this.plugin.settings.khojUrl}/api/update?t=pdf&force=true&client=obsidian`);
new Notice('✅ Updated Khoj index.');
// Reset button once index is updated
window.clearInterval(progress_indicator);
button.setButtonText('Update');
button.setCta()
button.setCta();
indexVaultSetting = indexVaultSetting.setDisabled(false);
})
);
@@ -76,7 +122,7 @@ export class KhojSettingTab extends PluginSettingTab {
getBackendStatusMessage() {
return !this.plugin.settings.connectedToBackend
? '❗Disconnected from Khoj backend. Ensure Khoj backend is running and Khoj URL is correctly set below.'
: '✅ Connected to Khoj backend.';
? '❗Disconnected from Khoj backend. Ensure Khoj backend is running and Khoj URL is correctly set below.'
: '✅ Connected to Khoj backend.';
}
}

199
src/interface/obsidian/src/utils.ts
View File

@@ -1,16 +1,18 @@
import { FileSystemAdapter, Notice, RequestUrlParam, request } from 'obsidian';
import { FileSystemAdapter, Notice, RequestUrlParam, request, Vault, Modal } from 'obsidian';
import { KhojSetting } from 'src/settings'
export function getVaultAbsolutePath(): string {
let adaptor = this.app.vault.adapter;
export function getVaultAbsolutePath(vault: Vault): string {
let adaptor = vault.adapter;
if (adaptor instanceof FileSystemAdapter) {
return adaptor.getBasePath();
}
return '';
}
export async function configureKhojBackend(setting: KhojSetting, notify: boolean = true) {
let mdInVault = `${getVaultAbsolutePath()}/**/*.md`;
export async function configureKhojBackend(vault: Vault, setting: KhojSetting, notify: boolean = true) {
let vaultPath = getVaultAbsolutePath(vault);
let mdInVault = `${vaultPath}/**/*.md`;
let pdfInVault = `${vaultPath}/**/*.pdf`;
let khojConfigUrl = `${setting.khojUrl}/api/config/data`;
// Check if khoj backend is configured, note if cannot connect to backend
@@ -28,11 +30,13 @@ export async function configureKhojBackend(setting: KhojSetting, notify: boolean
if (!setting.connectedToBackend) return;
// Set index name from the path of the current vault
let indexName = getVaultAbsolutePath().replace(/\//g, '_').replace(/ /g, '_');
// Get default index directory from khoj backend
let khojDefaultIndexDirectory = await request(`${khojConfigUrl}/default`)
.then(response => JSON.parse(response))
.then(data => { return getIndexDirectoryFromBackendConfig(data); });
let indexName = vaultPath.replace(/\//g, '_').replace(/\\/g, '_').replace(/ /g, '_').replace(/:/g, '_');
// Get default config fields from khoj backend
let defaultConfig = await request(`${khojConfigUrl}/default`).then(response => JSON.parse(response));
let khojDefaultMdIndexDirectory = getIndexDirectoryFromBackendConfig(defaultConfig["content-type"]["markdown"]["embeddings-file"]);
let khojDefaultPdfIndexDirectory = getIndexDirectoryFromBackendConfig(defaultConfig["content-type"]["pdf"]["embeddings-file"]);
let khojDefaultChatDirectory = getIndexDirectoryFromBackendConfig(defaultConfig["processor"]["conversation"]["conversation-logfile"]);
let khojDefaultChatModelName = defaultConfig["processor"]["conversation"]["model"];
// Get current config if khoj backend configured, else get default config from khoj backend
await request(khoj_already_configured ? khojConfigUrl : `${khojConfigUrl}/default`)
@@ -45,18 +49,22 @@ export async function configureKhojBackend(setting: KhojSetting, notify: boolean
"markdown": {
"input-filter": [mdInVault],
"input-files": null,
"embeddings-file": `${khojDefaultIndexDirectory}/${indexName}.pt`,
"compressed-jsonl": `${khojDefaultIndexDirectory}/${indexName}.jsonl.gz`,
"embeddings-file": `${khojDefaultMdIndexDirectory}/${indexName}.pt`,
"compressed-jsonl": `${khojDefaultMdIndexDirectory}/${indexName}.jsonl.gz`,
}
}
// Disable khoj processors, as not required
delete data["processor"];
// Save new config and refresh index on khoj backend
updateKhojBackend(setting.khojUrl, data);
console.log(`Khoj: Created khoj backend config:\n${JSON.stringify(data)}`)
const hasPdfFiles = app.vault.getFiles().some(file => file.extension === 'pdf');
if (hasPdfFiles) {
data["content-type"]["pdf"] = {
"input-filter": [pdfInVault],
"input-files": null,
"embeddings-file": `${khojDefaultPdfIndexDirectory}/${indexName}.pt`,
"compressed-jsonl": `${khojDefaultPdfIndexDirectory}/${indexName}.jsonl.gz`,
}
}
}
// Else if khoj config has no markdown content config
else if (!data["content-type"]["markdown"]) {
// Add markdown config to khoj content-type config
@@ -64,31 +72,104 @@ export async function configureKhojBackend(setting: KhojSetting, notify: boolean
data["content-type"]["markdown"] = {
"input-filter": [mdInVault],
"input-files": null,
"embeddings-file": `${khojDefaultIndexDirectory}/${indexName}.pt`,
"compressed-jsonl": `${khojDefaultIndexDirectory}/${indexName}.jsonl.gz`,
"embeddings-file": `${khojDefaultMdIndexDirectory}/${indexName}.pt`,
"compressed-jsonl": `${khojDefaultMdIndexDirectory}/${indexName}.jsonl.gz`,
}
// Save updated config and refresh index on khoj backend
updateKhojBackend(setting.khojUrl, data);
console.log(`Khoj: Added markdown config to khoj backend config:\n${JSON.stringify(data["content-type"])}`)
}
// Else if khoj is not configured to index markdown files in configured obsidian vault
else if (data["content-type"]["markdown"]["input-filter"].length != 1 ||
else if (
data["content-type"]["markdown"]["input-files"] != null ||
data["content-type"]["markdown"]["input-filter"] == null ||
data["content-type"]["markdown"]["input-filter"].length != 1 ||
data["content-type"]["markdown"]["input-filter"][0] !== mdInVault) {
// Update markdown config in khoj content-type config
// Set markdown config to only index markdown files in configured obsidian vault
let khojIndexDirectory = getIndexDirectoryFromBackendConfig(data);
data["content-type"]["markdown"] = {
"input-filter": [mdInVault],
"input-files": null,
"embeddings-file": `${khojIndexDirectory}/${indexName}.pt`,
"compressed-jsonl": `${khojIndexDirectory}/${indexName}.jsonl.gz`,
}
// Save updated config and refresh index on khoj backend
updateKhojBackend(setting.khojUrl, data);
console.log(`Khoj: Updated markdown config in khoj backend config:\n${JSON.stringify(data["content-type"]["markdown"])}`)
// Update markdown config in khoj content-type config
// Set markdown config to only index markdown files in configured obsidian vault
let khojMdIndexDirectory = getIndexDirectoryFromBackendConfig(data["content-type"]["markdown"]["embeddings-file"]);
data["content-type"]["markdown"] = {
"input-filter": [mdInVault],
"input-files": null,
"embeddings-file": `${khojMdIndexDirectory}/${indexName}.pt`,
"compressed-jsonl": `${khojMdIndexDirectory}/${indexName}.jsonl.gz`,
}
}
if (khoj_already_configured && !data["content-type"]["pdf"]) {
const hasPdfFiles = app.vault.getFiles().some(file => file.extension === 'pdf');
if (hasPdfFiles) {
data["content-type"]["pdf"] = {
"input-filter": [pdfInVault],
"input-files": null,
"embeddings-file": `${khojDefaultPdfIndexDirectory}/${indexName}.pt`,
"compressed-jsonl": `${khojDefaultPdfIndexDirectory}/${indexName}.jsonl.gz`,
}
} else {
data["content-type"]["pdf"] = null;
}
}
// Else if khoj is not configured to index pdf files in configured obsidian vault
else if (khoj_already_configured &&
(
data["content-type"]["pdf"]["input-files"] != null ||
data["content-type"]["pdf"]["input-filter"] == null ||
data["content-type"]["pdf"]["input-filter"].length != 1 ||
data["content-type"]["pdf"]["input-filter"][0] !== pdfInVault)) {
let hasPdfFiles = app.vault.getFiles().some(file => file.extension === 'pdf');
if (hasPdfFiles) {
// Update pdf config in khoj content-type config
// Set pdf config to only index pdf files in configured obsidian vault
let khojPdfIndexDirectory = getIndexDirectoryFromBackendConfig(data["content-type"]["pdf"]["embeddings-file"]);
data["content-type"]["pdf"] = {
"input-filter": [pdfInVault],
"input-files": null,
"embeddings-file": `${khojPdfIndexDirectory}/${indexName}.pt`,
"compressed-jsonl": `${khojPdfIndexDirectory}/${indexName}.jsonl.gz`,
}
} else {
data["content-type"]["pdf"] = null;
}
}
// If OpenAI API key not set in Khoj plugin settings
if (!setting.openaiApiKey) {
// Disable khoj processors, as not required
delete data["processor"];
}
// Else if khoj backend not configured yet
else if (!khoj_already_configured || !data["processor"]) {
data["processor"] = {
"conversation": {
"conversation-logfile": `${khojDefaultChatDirectory}/conversation.json`,
"model": khojDefaultChatModelName,
"openai-api-key": setting.openaiApiKey,
}
}
}
// Else if khoj config has no conversation processor config
else if (!data["processor"]["conversation"]) {
data["processor"]["conversation"] = {
"conversation-logfile": `${khojDefaultChatDirectory}/conversation.json`,
"model": khojDefaultChatModelName,
"openai-api-key": setting.openaiApiKey,
}
}
// Else if khoj is not configured with OpenAI API key from khoj plugin settings
else if (data["processor"]["conversation"]["openai-api-key"] !== setting.openaiApiKey) {
data["processor"]["conversation"] = {
"conversation-logfile": data["processor"]["conversation"]["conversation-logfile"],
"model": data["processor"]["conversation"]["model"],
"openai-api-key": setting.openaiApiKey,
}
}
// Save updated config and refresh index on khoj backend
updateKhojBackend(setting.khojUrl, data);
if (!khoj_already_configured)
console.log(`Khoj: Created khoj backend config:\n${JSON.stringify(data)}`)
else
console.log(`Khoj: Updated khoj backend config:\n${JSON.stringify(data)}`)
})
.catch(error => {
if (notify)
@@ -108,9 +189,43 @@ export async function updateKhojBackend(khojUrl: string, khojConfig: Object) {
// Save khojConfig on khoj backend at khojConfigUrl
await request(requestContent)
// Refresh khoj search index after updating config
.then(_ => request(`${khojUrl}/api/update?t=markdown`));
.then(_ => request(`${khojUrl}/api/update?t=markdown`))
.then(_ => request(`${khojUrl}/api/update?t=pdf`));
}
function getIndexDirectoryFromBackendConfig(khojConfig: any) {
return khojConfig["content-type"]["markdown"]["embeddings-file"].split("/").slice(0, -1).join("/");
}
function getIndexDirectoryFromBackendConfig(filepath: string) {
return filepath.split("/").slice(0, -1).join("/");
}
export async function createNote(name: string, newLeaf = false): Promise<void> {
try {
let pathPrefix: string
// @ts-ignore
switch (app.vault.getConfig('newFileLocation')) {
case 'current':
pathPrefix = (app.workspace.getActiveFile()?.parent.path ?? '') + '/'
break
case 'folder':
pathPrefix = this.app.vault.getConfig('newFileFolderPath') + '/'
break
default: // 'root'
pathPrefix = ''
break
}
await app.workspace.openLinkText(`${pathPrefix}${name}.md`, '', newLeaf)
} catch (e) {
console.error('Khoj: Could not create note.\n' + (e as any).message);
throw e
}
}
export async function createNoteAndCloseModal(query: string, modal: Modal, opt?: { newLeaf: boolean }): Promise<void> {
try {
await createNote(query, opt?.newLeaf);
}
catch (e) {
new Notice((e as Error).message)
return
}
modal.close();
}

176
src/interface/obsidian/styles.css
View File

@@ -6,3 +6,179 @@ available in the app when your plugin is enabled.
If your plugin does not need CSS, delete this file.
*/
:root {
--khoj-chat-primary: #ffb300;
--khoj-chat-dark-grey: #475569;
}
.khoj-chat {
display: grid;
background: var(--background-primary);
color: var(--text-normal);
text-align: center;
font-family: roboto, karma, segoe ui, sans-serif;
font-size: var(--font-ui-large);
font-weight: 300;
line-height: 1.5em;
}
.khoj-chat > * {
padding: 10px;
margin: 10px;
}
#khoj-chat-title {
font-weight: 200;
color: var(--khoj-chat-primary);
}
#khoj-chat-body {
font-size: var(--font-ui-medium);
margin: 0px;
line-height: 20px;
overflow-y: scroll; /* Make chat body scroll to see history */
}
/* add chat metatdata to bottom of bubble */
.khoj-chat-message::after {
content: attr(data-meta);
display: block;
font-size: var(--font-ui-smaller);
color: var(--text-muted);
margin: -12px 7px 0 -5px;
}
/* move message by khoj to left */
.khoj-chat-message.khoj {
margin-left: auto;
text-align: left;
}
/* move message by you to right */
.khoj-chat-message.you {
margin-right: auto;
text-align: right;
}
/* basic style chat message text */
.khoj-chat-message-text {
margin: 10px;
border-radius: 10px;
padding: 10px;
position: relative;
display: inline-block;
max-width: 80%;
text-align: left;
}
/* color chat bubble by khoj blue */
.khoj-chat-message-text.khoj {
color: var(--text-on-accent);
background: var(--khoj-chat-primary);
margin-left: auto;
white-space: pre-line;
}
/* add left protrusion to khoj chat bubble */
.khoj-chat-message-text.khoj:after {
content: '';
position: absolute;
bottom: -2px;
left: -7px;
border: 10px solid transparent;
border-top-color: var(--khoj-chat-primary);
border-bottom: 0;
transform: rotate(-60deg);
}
/* color chat bubble by you dark grey */
.khoj-chat-message-text.you {
color: var(--text-on-accent);
background: var(--khoj-chat-dark-grey);
margin-right: auto;
}
/* add right protrusion to you chat bubble */
.khoj-chat-message-text.you:after {
content: '';
position: absolute;
top: 91%;
right: -2px;
border: 10px solid transparent;
border-left-color: var(--khoj-chat-dark-grey);
border-right: 0;
margin-top: -10px;
transform: rotate(-60deg)
}
#khoj-chat-footer {
padding: 0;
display: grid;
grid-template-columns: minmax(70px, 100%);
grid-column-gap: 10px;
grid-row-gap: 10px;
}
#khoj-chat-footer > * {
padding: 15px;
background: #f9fafc
}
#khoj-chat-input.option:hover {
box-shadow: 0 0 11px var(--background-modifier-box-shadow);
}
#khoj-chat-input {
font-size: var(--font-ui-medium);
padding: 25px 20px;
}
@media (pointer: coarse), (hover: none) {
#khoj-chat-body.abbr[title] {
position: relative;
padding-left: 4px; /* space references out to ease tapping */
}
#khoj-chat-body.abbr[title]:focus:after {
content: attr(title);
/* position tooltip */
position: absolute;
left: 16px; /* open tooltip to right of ref link, instead of on top of it */
width: auto;
z-index: 1; /* show tooltip above chat messages */
/* style tooltip */
background-color: var(--background-secondary);
color: var(--text-muted);
border-radius: 2px;
box-shadow: 1px 1px 4px 0 var(--background-modifier-box-shadow);
font-size: var(--font-ui-small);
padding: 2px 4px;
}
}
.khoj-result-file {
font-weight: 600;
}
.khoj-result-entry {
color: var(--text-muted);
margin-left: 2em;
padding-left: 0.5em;
line-height: normal;
margin-top: 0.2em;
margin-bottom: 0.2em;
border-left-style: solid;
border-left-color: var(--color-accent-2);
white-space: normal;
}
.khoj-result-entry > * {
font-size: var(--font-ui-medium);
}
.khoj-result-entry > p {
margin-top: 0.2em;
margin-bottom: 0.2em;
}
.khoj-result-entry p br {
display: none;
}
.khoj-info-hint {
color: var(--text-muted);
font-size: var(--font-ui-small);
font-style: italic;
text-align: center;
margin-bottom: 0.5em;
}

14
src/interface/obsidian/versions.json
View File

@@ -1,4 +1,14 @@
{
"0.2.1": "0.15.0",
"0.2.5": "0.15.0"
"0.2.1": "0.15.0",
"0.2.5": "0.15.0",
"0.2.6": "0.15.0",
"0.3.0": "0.15.0",
"0.4.0": "0.15.0",
"0.5.0": "0.15.0",
"0.6.0": "0.15.0",
"0.6.1": "0.15.0",
"0.6.2": "0.15.0",
"0.7.0": "0.15.0",
"0.7.1": "0.15.0",
"0.8.0": "0.15.0"
}

609
src/interface/obsidian/yarn.lock Normal file
View File

@@ -0,0 +1,609 @@
# THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY.
# yarn lockfile v1
"@nodelib/fs.scandir@2.1.5":
version "2.1.5"
resolved "https://registry.npmjs.org/@nodelib/fs.scandir/-/fs.scandir-2.1.5.tgz"
integrity sha512-vq24Bq3ym5HEQm2NKCr3yXDwjc7vTsEThRDnkp2DK9p1uqLR+DHurm/NOTo0KG7HYHU7eppKZj3MyqYuMBf62g==
dependencies:
"@nodelib/fs.stat" "2.0.5"
run-parallel "^1.1.9"
"@nodelib/fs.stat@2.0.5", "@nodelib/fs.stat@^2.0.2":
version "2.0.5"
resolved "https://registry.npmjs.org/@nodelib/fs.stat/-/fs.stat-2.0.5.tgz"
integrity sha512-RkhPPp2zrqDAQA/2jNhnztcPAlv64XdhIp7a7454A5ovI7Bukxgt7MX7udwAu3zg1DcpPU0rz3VV1SeaqvY4+A==
"@nodelib/fs.walk@^1.2.3":
version "1.2.8"
resolved "https://registry.npmjs.org/@nodelib/fs.walk/-/fs.walk-1.2.8.tgz"
integrity sha512-oGB+UxlgWcgQkgwo8GcEGwemoTFt3FIO9ababBmaGwXIoBKZ+GTy0pP185beGg7Llih/NSHSV2XAs1lnznocSg==
dependencies:
"@nodelib/fs.scandir" "2.1.5"
fastq "^1.6.0"
"@types/codemirror@0.0.108":
version "0.0.108"
resolved "https://registry.npmjs.org/@types/codemirror/-/codemirror-0.0.108.tgz"
integrity sha512-3FGFcus0P7C2UOGCNUVENqObEb4SFk+S8Dnxq7K6aIsLVs/vDtlangl3PEO0ykaKXyK56swVF6Nho7VsA44uhw==
dependencies:
"@types/tern" "*"
"@types/estree@*":
version "1.0.0"
resolved "https://registry.npmjs.org/@types/estree/-/estree-1.0.0.tgz"
integrity sha512-WulqXMDUTYAXCjZnk6JtIHPigp55cVtDgDrO2gHRwhyJto21+1zbVCtOYB2L1F9w4qCQ0rOGWBnBe0FNTiEJIQ==
"@types/json-schema@^7.0.9":
version "7.0.11"
resolved "https://registry.npmjs.org/@types/json-schema/-/json-schema-7.0.11.tgz"
integrity sha512-wOuvG1SN4Us4rez+tylwwwCV1psiNVOkJeM3AUWUNWg/jDQY2+HE/444y5gc+jBmRqASOm2Oeh5c1axHobwRKQ==
"@types/node-fetch@^2.6.4":
version "2.6.4"
resolved "https://registry.yarnpkg.com/@types/node-fetch/-/node-fetch-2.6.4.tgz#1bc3a26de814f6bf466b25aeb1473fa1afe6a660"
integrity sha512-1ZX9fcN4Rvkvgv4E6PAY5WXUFWFcRWxZa3EW83UjycOB9ljJCedb2CupIP4RZMEwF/M3eTcCihbBRgwtGbg5Rg==
dependencies:
"@types/node" "*"
form-data "^3.0.0"
"@types/node@*":
version "20.3.3"
resolved "https://registry.yarnpkg.com/@types/node/-/node-20.3.3.tgz#329842940042d2b280897150e023e604d11657d6"
integrity sha512-wheIYdr4NYML61AjC8MKj/2jrR/kDQri/CIpVoZwldwhnIrD/j9jIU5bJ8yBKuB2VhpFV7Ab6G2XkBjv9r9Zzw==
"@types/node@^16.11.6":
version "16.18.12"
resolved "https://registry.npmjs.org/@types/node/-/node-16.18.12.tgz"
integrity sha512-vzLe5NaNMjIE3mcddFVGlAXN1LEWueUsMsOJWaT6wWMJGyljHAWHznqfnKUQWGzu7TLPrGvWdNAsvQYW+C0xtw==
"@types/tern@*":
version "0.23.4"
resolved "https://registry.npmjs.org/@types/tern/-/tern-0.23.4.tgz"
integrity sha512-JAUw1iXGO1qaWwEOzxTKJZ/5JxVeON9kvGZ/osgZaJImBnyjyn0cjovPsf6FNLmyGY8Vw9DoXZCMlfMkMwHRWg==
dependencies:
"@types/estree" "*"
"@typescript-eslint/eslint-plugin@5.29.0":
version "5.29.0"
resolved "https://registry.npmjs.org/@typescript-eslint/eslint-plugin/-/eslint-plugin-5.29.0.tgz"
integrity sha512-kgTsISt9pM53yRFQmLZ4npj99yGl3x3Pl7z4eA66OuTzAGC4bQB5H5fuLwPnqTKU3yyrrg4MIhjF17UYnL4c0w==
dependencies:
"@typescript-eslint/scope-manager" "5.29.0"
"@typescript-eslint/type-utils" "5.29.0"
"@typescript-eslint/utils" "5.29.0"
debug "^4.3.4"
functional-red-black-tree "^1.0.1"
ignore "^5.2.0"
regexpp "^3.2.0"
semver "^7.3.7"
tsutils "^3.21.0"
"@typescript-eslint/parser@5.29.0":
version "5.29.0"
resolved "https://registry.npmjs.org/@typescript-eslint/parser/-/parser-5.29.0.tgz"
integrity sha512-ruKWTv+x0OOxbzIw9nW5oWlUopvP/IQDjB5ZqmTglLIoDTctLlAJpAQFpNPJP/ZI7hTT9sARBosEfaKbcFuECw==
dependencies:
"@typescript-eslint/scope-manager" "5.29.0"
"@typescript-eslint/types" "5.29.0"
"@typescript-eslint/typescript-estree" "5.29.0"
debug "^4.3.4"
"@typescript-eslint/scope-manager@5.29.0":
version "5.29.0"
resolved "https://registry.npmjs.org/@typescript-eslint/scope-manager/-/scope-manager-5.29.0.tgz"
integrity sha512-etbXUT0FygFi2ihcxDZjz21LtC+Eps9V2xVx09zFoN44RRHPrkMflidGMI+2dUs821zR1tDS6Oc9IXxIjOUZwA==
dependencies:
"@typescript-eslint/types" "5.29.0"
"@typescript-eslint/visitor-keys" "5.29.0"
"@typescript-eslint/type-utils@5.29.0":
version "5.29.0"
resolved "https://registry.npmjs.org/@typescript-eslint/type-utils/-/type-utils-5.29.0.tgz"
integrity sha512-JK6bAaaiJozbox3K220VRfCzLa9n0ib/J+FHIwnaV3Enw/TO267qe0pM1b1QrrEuy6xun374XEAsRlA86JJnyg==
dependencies:
"@typescript-eslint/utils" "5.29.0"
debug "^4.3.4"
tsutils "^3.21.0"
"@typescript-eslint/types@5.29.0":
version "5.29.0"
resolved "https://registry.npmjs.org/@typescript-eslint/types/-/types-5.29.0.tgz"
integrity sha512-X99VbqvAXOMdVyfFmksMy3u8p8yoRGITgU1joBJPzeYa0rhdf5ok9S56/itRoUSh99fiDoMtarSIJXo7H/SnOg==
"@typescript-eslint/typescript-estree@5.29.0":
version "5.29.0"
resolved "https://registry.npmjs.org/@typescript-eslint/typescript-estree/-/typescript-estree-5.29.0.tgz"
integrity sha512-mQvSUJ/JjGBdvo+1LwC+GY2XmSYjK1nAaVw2emp/E61wEVYEyibRHCqm1I1vEKbXCpUKuW4G7u9ZCaZhJbLoNQ==
dependencies:
"@typescript-eslint/types" "5.29.0"
"@typescript-eslint/visitor-keys" "5.29.0"
debug "^4.3.4"
globby "^11.1.0"
is-glob "^4.0.3"
semver "^7.3.7"
tsutils "^3.21.0"
"@typescript-eslint/utils@5.29.0":
version "5.29.0"
resolved "https://registry.npmjs.org/@typescript-eslint/utils/-/utils-5.29.0.tgz"
integrity sha512-3Eos6uP1nyLOBayc/VUdKZikV90HahXE5Dx9L5YlSd/7ylQPXhLk1BYb29SDgnBnTp+jmSZUU0QxUiyHgW4p7A==
dependencies:
"@types/json-schema" "^7.0.9"
"@typescript-eslint/scope-manager" "5.29.0"
"@typescript-eslint/types" "5.29.0"
"@typescript-eslint/typescript-estree" "5.29.0"
eslint-scope "^5.1.1"
eslint-utils "^3.0.0"
"@typescript-eslint/visitor-keys@5.29.0":
version "5.29.0"
resolved "https://registry.npmjs.org/@typescript-eslint/visitor-keys/-/visitor-keys-5.29.0.tgz"
integrity sha512-Hpb/mCWsjILvikMQoZIE3voc9wtQcS0A9FUw3h8bhr9UxBdtI/tw1ZDZUOXHXLOVMedKCH5NxyzATwnU78bWCQ==
dependencies:
"@typescript-eslint/types" "5.29.0"
eslint-visitor-keys "^3.3.0"
array-union@^2.1.0:
version "2.1.0"
resolved "https://registry.npmjs.org/array-union/-/array-union-2.1.0.tgz"
integrity sha512-HGyxoOTYUyCM6stUe6EJgnd4EoewAI7zMdfqO+kGjnlZmBDz/cR5pf8r/cR4Wq60sL/p0IkcjUEEPwS3GFrIyw==
asynckit@^0.4.0:
version "0.4.0"
resolved "https://registry.yarnpkg.com/asynckit/-/asynckit-0.4.0.tgz#c79ed97f7f34cb8f2ba1bc9790bcc366474b4b79"
integrity sha512-Oei9OH4tRh0YqU3GxhX79dM/mwVgvbZJaSNaRk+bshkj0S5cfHcgYakreBjrHwatXKbz+IoIdYLxrKim2MjW0Q==
braces@^3.0.2:
version "3.0.2"
resolved "https://registry.npmjs.org/braces/-/braces-3.0.2.tgz"
integrity sha512-b8um+L1RzM3WDSzvhm6gIz1yfTbBt6YTlcEKAvsmqCZZFw46z626lVj9j1yEPW33H5H+lBQpZMP1k8l+78Ha0A==
dependencies:
fill-range "^7.0.1"
builtin-modules@3.3.0:
version "3.3.0"
resolved "https://registry.npmjs.org/builtin-modules/-/builtin-modules-3.3.0.tgz"
integrity sha512-zhaCDicdLuWN5UbN5IMnFqNMhNfo919sH85y2/ea+5Yg9TsTkeZxpL+JLbp6cgYFS4sRLp3YV4S6yDuqVWHYOw==
combined-stream@^1.0.8:
version "1.0.8"
resolved "https://registry.yarnpkg.com/combined-stream/-/combined-stream-1.0.8.tgz#c3d45a8b34fd730631a110a8a2520682b31d5a7f"
integrity sha512-FQN4MRfuJeHf7cBbBMJFXhKSDq+2kAArBlmRBvcvFE5BB1HZKXtSFASDhdlz9zOYwxh8lDdnvmMOe/+5cdoEdg==
dependencies:
delayed-stream "~1.0.0"
data-uri-to-buffer@^4.0.0:
version "4.0.1"
resolved "https://registry.yarnpkg.com/data-uri-to-buffer/-/data-uri-to-buffer-4.0.1.tgz#d8feb2b2881e6a4f58c2e08acfd0e2834e26222e"
integrity sha512-0R9ikRb668HB7QDxT1vkpuUBtqc53YyAwMwGeUFKRojY/NWKvdZ+9UYtRfGmhqNbRkTSVpMbmyhXipFFv2cb/A==
debug@^4.3.4:
version "4.3.4"
resolved "https://registry.npmjs.org/debug/-/debug-4.3.4.tgz"
integrity sha512-PRWFHuSU3eDtQJPvnNY7Jcket1j0t5OuOsFzPPzsekD52Zl8qUfFIPEiswXqIvHWGVHOgX+7G/vCNNhehwxfkQ==
dependencies:
ms "2.1.2"
delayed-stream@~1.0.0:
version "1.0.0"
resolved "https://registry.yarnpkg.com/delayed-stream/-/delayed-stream-1.0.0.tgz#df3ae199acadfb7d440aaae0b29e2272b24ec619"
integrity sha512-ZySD7Nf91aLB0RxL4KGrKHBXl7Eds1DAmEdcoVawXnLD7SDhpNgtuII2aAkg7a7QS41jxPSZ17p4VdGnMHk3MQ==
dir-glob@^3.0.1:
version "3.0.1"
resolved "https://registry.npmjs.org/dir-glob/-/dir-glob-3.0.1.tgz"
integrity sha512-WkrWp9GR4KXfKGYzOLmTuGVi1UWFfws377n9cc55/tb6DuqyF6pcQ5AbiHEshaDpY9v6oaSr2XCDidGmMwdzIA==
dependencies:
path-type "^4.0.0"
esbuild-android-64@0.14.47:
version "0.14.47"
resolved "https://registry.yarnpkg.com/esbuild-android-64/-/esbuild-android-64-0.14.47.tgz#ef95b42c67bcf4268c869153fa3ad1466c4cea6b"
integrity sha512-R13Bd9+tqLVFndncMHssZrPWe6/0Kpv2/dt4aA69soX4PRxlzsVpCvoJeFE8sOEoeVEiBkI0myjlkDodXlHa0g==
esbuild-android-arm64@0.14.47:
version "0.14.47"
resolved "https://registry.yarnpkg.com/esbuild-android-arm64/-/esbuild-android-arm64-0.14.47.tgz#4ebd7ce9fb250b4695faa3ee46fd3b0754ecd9e6"
integrity sha512-OkwOjj7ts4lBp/TL6hdd8HftIzOy/pdtbrNA4+0oVWgGG64HrdVzAF5gxtJufAPOsEjkyh1oIYvKAUinKKQRSQ==
esbuild-darwin-64@0.14.47:
version "0.14.47"
resolved "https://registry.yarnpkg.com/esbuild-darwin-64/-/esbuild-darwin-64-0.14.47.tgz#e0da6c244f497192f951807f003f6a423ed23188"
integrity sha512-R6oaW0y5/u6Eccti/TS6c/2c1xYTb1izwK3gajJwi4vIfNs1s8B1dQzI1UiC9T61YovOQVuePDcfqHLT3mUZJA==
esbuild-darwin-arm64@0.14.47:
version "0.14.47"
resolved "https://registry.npmjs.org/esbuild-darwin-arm64/-/esbuild-darwin-arm64-0.14.47.tgz"
integrity sha512-seCmearlQyvdvM/noz1L9+qblC5vcBrhUaOoLEDDoLInF/VQ9IkobGiLlyTPYP5dW1YD4LXhtBgOyevoIHGGnw==
esbuild-freebsd-64@0.14.47:
version "0.14.47"
resolved "https://registry.yarnpkg.com/esbuild-freebsd-64/-/esbuild-freebsd-64-0.14.47.tgz#8da6a14c095b29c01fc8087a16cb7906debc2d67"
integrity sha512-ZH8K2Q8/Ux5kXXvQMDsJcxvkIwut69KVrYQhza/ptkW50DC089bCVrJZZ3sKzIoOx+YPTrmsZvqeZERjyYrlvQ==
esbuild-freebsd-arm64@0.14.47:
version "0.14.47"
resolved "https://registry.yarnpkg.com/esbuild-freebsd-arm64/-/esbuild-freebsd-arm64-0.14.47.tgz#ad31f9c92817ff8f33fd253af7ab5122dc1b83f6"
integrity sha512-ZJMQAJQsIOhn3XTm7MPQfCzEu5b9STNC+s90zMWe2afy9EwnHV7Ov7ohEMv2lyWlc2pjqLW8QJnz2r0KZmeAEQ==
esbuild-linux-32@0.14.47:
version "0.14.47"
resolved "https://registry.yarnpkg.com/esbuild-linux-32/-/esbuild-linux-32-0.14.47.tgz#de085e4db2e692ea30c71208ccc23fdcf5196c58"
integrity sha512-FxZOCKoEDPRYvq300lsWCTv1kcHgiiZfNrPtEhFAiqD7QZaXrad8LxyJ8fXGcWzIFzRiYZVtB3ttvITBvAFhKw==
esbuild-linux-64@0.14.47:
version "0.14.47"
resolved "https://registry.yarnpkg.com/esbuild-linux-64/-/esbuild-linux-64-0.14.47.tgz#2a9321bbccb01f01b04cebfcfccbabeba3658ba1"
integrity sha512-nFNOk9vWVfvWYF9YNYksZptgQAdstnDCMtR6m42l5Wfugbzu11VpMCY9XrD4yFxvPo9zmzcoUL/88y0lfJZJJw==
esbuild-linux-arm64@0.14.47:
version "0.14.47"
resolved "https://registry.yarnpkg.com/esbuild-linux-arm64/-/esbuild-linux-arm64-0.14.47.tgz#b9da7b6fc4b0ca7a13363a0c5b7bb927e4bc535a"
integrity sha512-ywfme6HVrhWcevzmsufjd4iT3PxTfCX9HOdxA7Hd+/ZM23Y9nXeb+vG6AyA6jgq/JovkcqRHcL9XwRNpWG6XRw==
esbuild-linux-arm@0.14.47:
version "0.14.47"
resolved "https://registry.yarnpkg.com/esbuild-linux-arm/-/esbuild-linux-arm-0.14.47.tgz#56fec2a09b9561c337059d4af53625142aded853"
integrity sha512-ZGE1Bqg/gPRXrBpgpvH81tQHpiaGxa8c9Rx/XOylkIl2ypLuOcawXEAo8ls+5DFCcRGt/o3sV+PzpAFZobOsmA==
esbuild-linux-mips64le@0.14.47:
version "0.14.47"
resolved "https://registry.yarnpkg.com/esbuild-linux-mips64le/-/esbuild-linux-mips64le-0.14.47.tgz#9db21561f8f22ed79ef2aedb7bbef082b46cf823"
integrity sha512-mg3D8YndZ1LvUiEdDYR3OsmeyAew4MA/dvaEJxvyygahWmpv1SlEEnhEZlhPokjsUMfRagzsEF/d/2XF+kTQGg==
esbuild-linux-ppc64le@0.14.47:
version "0.14.47"
resolved "https://registry.yarnpkg.com/esbuild-linux-ppc64le/-/esbuild-linux-ppc64le-0.14.47.tgz#dc3a3da321222b11e96e50efafec9d2de408198b"
integrity sha512-WER+f3+szmnZiWoK6AsrTKGoJoErG2LlauSmk73LEZFQ/iWC+KhhDsOkn1xBUpzXWsxN9THmQFltLoaFEH8F8w==
esbuild-linux-riscv64@0.14.47:
version "0.14.47"
resolved "https://registry.yarnpkg.com/esbuild-linux-riscv64/-/esbuild-linux-riscv64-0.14.47.tgz#9bd6dcd3dca6c0357084ecd06e1d2d4bf105335f"
integrity sha512-1fI6bP3A3rvI9BsaaXbMoaOjLE3lVkJtLxsgLHqlBhLlBVY7UqffWBvkrX/9zfPhhVMd9ZRFiaqXnB1T7BsL2g==
esbuild-linux-s390x@0.14.47:
version "0.14.47"
resolved "https://registry.yarnpkg.com/esbuild-linux-s390x/-/esbuild-linux-s390x-0.14.47.tgz#a458af939b52f2cd32fc561410d441a51f69d41f"
integrity sha512-eZrWzy0xFAhki1CWRGnhsHVz7IlSKX6yT2tj2Eg8lhAwlRE5E96Hsb0M1mPSE1dHGpt1QVwwVivXIAacF/G6mw==
esbuild-netbsd-64@0.14.47:
version "0.14.47"
resolved "https://registry.yarnpkg.com/esbuild-netbsd-64/-/esbuild-netbsd-64-0.14.47.tgz#6388e785d7e7e4420cb01348d7483ab511b16aa8"
integrity sha512-Qjdjr+KQQVH5Q2Q1r6HBYswFTToPpss3gqCiSw2Fpq/ua8+eXSQyAMG+UvULPqXceOwpnPo4smyZyHdlkcPppQ==
esbuild-openbsd-64@0.14.47:
version "0.14.47"
resolved "https://registry.yarnpkg.com/esbuild-openbsd-64/-/esbuild-openbsd-64-0.14.47.tgz#309af806db561aa886c445344d1aacab850dbdc5"
integrity sha512-QpgN8ofL7B9z8g5zZqJE+eFvD1LehRlxr25PBkjyyasakm4599iroUpaj96rdqRlO2ShuyqwJdr+oNqWwTUmQw==
esbuild-sunos-64@0.14.47:
version "0.14.47"
resolved "https://registry.yarnpkg.com/esbuild-sunos-64/-/esbuild-sunos-64-0.14.47.tgz#3f19612dcdb89ba6c65283a7ff6e16f8afbf8aaa"
integrity sha512-uOeSgLUwukLioAJOiGYm3kNl+1wJjgJA8R671GYgcPgCx7QR73zfvYqXFFcIO93/nBdIbt5hd8RItqbbf3HtAQ==
esbuild-windows-32@0.14.47:
version "0.14.47"
resolved "https://registry.yarnpkg.com/esbuild-windows-32/-/esbuild-windows-32-0.14.47.tgz#a92d279c8458d5dc319abcfeb30aa49e8f2e6f7f"
integrity sha512-H0fWsLTp2WBfKLBgwYT4OTfFly4Im/8B5f3ojDv1Kx//kiubVY0IQunP2Koc/fr/0wI7hj3IiBDbSrmKlrNgLQ==
esbuild-windows-64@0.14.47:
version "0.14.47"
resolved "https://registry.yarnpkg.com/esbuild-windows-64/-/esbuild-windows-64-0.14.47.tgz#2564c3fcf0c23d701edb71af8c52d3be4cec5f8a"
integrity sha512-/Pk5jIEH34T68r8PweKRi77W49KwanZ8X6lr3vDAtOlH5EumPE4pBHqkCUdELanvsT14yMXLQ/C/8XPi1pAtkQ==
esbuild-windows-arm64@0.14.47:
version "0.14.47"
resolved "https://registry.yarnpkg.com/esbuild-windows-arm64/-/esbuild-windows-arm64-0.14.47.tgz#86d9db1a22d83360f726ac5fba41c2f625db6878"
integrity sha512-HFSW2lnp62fl86/qPQlqw6asIwCnEsEoNIL1h2uVMgakddf+vUuMcCbtUY1i8sst7KkgHrVKCJQB33YhhOweCQ==
esbuild@0.14.47:
version "0.14.47"
resolved "https://registry.npmjs.org/esbuild/-/esbuild-0.14.47.tgz"
integrity sha512-wI4ZiIfFxpkuxB8ju4MHrGwGLyp1+awEHAHVpx6w7a+1pmYIq8T9FGEVVwFo0iFierDoMj++Xq69GXWYn2EiwA==
optionalDependencies:
esbuild-android-64 "0.14.47"
esbuild-android-arm64 "0.14.47"
esbuild-darwin-64 "0.14.47"
esbuild-darwin-arm64 "0.14.47"
esbuild-freebsd-64 "0.14.47"
esbuild-freebsd-arm64 "0.14.47"
esbuild-linux-32 "0.14.47"
esbuild-linux-64 "0.14.47"
esbuild-linux-arm "0.14.47"
esbuild-linux-arm64 "0.14.47"
esbuild-linux-mips64le "0.14.47"
esbuild-linux-ppc64le "0.14.47"
esbuild-linux-riscv64 "0.14.47"
esbuild-linux-s390x "0.14.47"
esbuild-netbsd-64 "0.14.47"
esbuild-openbsd-64 "0.14.47"
esbuild-sunos-64 "0.14.47"
esbuild-windows-32 "0.14.47"
esbuild-windows-64 "0.14.47"
esbuild-windows-arm64 "0.14.47"
eslint-scope@^5.1.1:
version "5.1.1"
resolved "https://registry.npmjs.org/eslint-scope/-/eslint-scope-5.1.1.tgz"
integrity sha512-2NxwbF/hZ0KpepYN0cNbo+FN6XoK7GaHlQhgx/hIZl6Va0bF45RQOOwhLIy8lQDbuCiadSLCBnH2CFYquit5bw==
dependencies:
esrecurse "^4.3.0"
estraverse "^4.1.1"
eslint-utils@^3.0.0:
version "3.0.0"
resolved "https://registry.npmjs.org/eslint-utils/-/eslint-utils-3.0.0.tgz"
integrity sha512-uuQC43IGctw68pJA1RgbQS8/NP7rch6Cwd4j3ZBtgo4/8Flj4eGE7ZYSZRN3iq5pVUv6GPdW5Z1RFleo84uLDA==
dependencies:
eslint-visitor-keys "^2.0.0"
eslint-visitor-keys@^2.0.0:
version "2.1.0"
resolved "https://registry.npmjs.org/eslint-visitor-keys/-/eslint-visitor-keys-2.1.0.tgz"
integrity sha512-0rSmRBzXgDzIsD6mGdJgevzgezI534Cer5L/vyMX0kHzT/jiB43jRhd9YUlMGYLQy2zprNmoT8qasCGtY+QaKw==
eslint-visitor-keys@^3.3.0:
version "3.3.0"
resolved "https://registry.npmjs.org/eslint-visitor-keys/-/eslint-visitor-keys-3.3.0.tgz"
integrity sha512-mQ+suqKJVyeuwGYHAdjMFqjCyfl8+Ldnxuyp3ldiMBFKkvytrXUZWaiPCEav8qDHKty44bD+qV1IP4T+w+xXRA==
esrecurse@^4.3.0:
version "4.3.0"
resolved "https://registry.npmjs.org/esrecurse/-/esrecurse-4.3.0.tgz"
integrity sha512-KmfKL3b6G+RXvP8N1vr3Tq1kL/oCFgn2NYXEtqP8/L3pKapUA4G8cFVaoF3SU323CD4XypR/ffioHmkti6/Tag==
dependencies:
estraverse "^5.2.0"
estraverse@^4.1.1:
version "4.3.0"
resolved "https://registry.npmjs.org/estraverse/-/estraverse-4.3.0.tgz"
integrity sha512-39nnKffWz8xN1BU/2c79n9nB9HDzo0niYUqx6xyqUnyoAnQyyWpOTdZEeiCch8BBu515t4wp9ZmgVfVhn9EBpw==
estraverse@^5.2.0:
version "5.3.0"
resolved "https://registry.npmjs.org/estraverse/-/estraverse-5.3.0.tgz"
integrity sha512-MMdARuVEQziNTeJD8DgMqmhwR11BRQ/cBP+pLtYdSTnf3MIO8fFeiINEbX36ZdNlfU/7A9f3gUw49B3oQsvwBA==
fast-glob@^3.2.9:
version "3.2.12"
resolved "https://registry.npmjs.org/fast-glob/-/fast-glob-3.2.12.tgz"
integrity sha512-DVj4CQIYYow0BlaelwK1pHl5n5cRSJfM60UA0zK891sVInoPri2Ekj7+e1CT3/3qxXenpI+nBBmQAcJPJgaj4w==
dependencies:
"@nodelib/fs.stat" "^2.0.2"
"@nodelib/fs.walk" "^1.2.3"
glob-parent "^5.1.2"
merge2 "^1.3.0"
micromatch "^4.0.4"
fastq@^1.6.0:
version "1.15.0"
resolved "https://registry.npmjs.org/fastq/-/fastq-1.15.0.tgz"
integrity sha512-wBrocU2LCXXa+lWBt8RoIRD89Fi8OdABODa/kEnyeyjS5aZO5/GNvI5sEINADqP/h8M29UHTHUb53sUu5Ihqdw==
dependencies:
reusify "^1.0.4"
fetch-blob@^3.1.2, fetch-blob@^3.1.4:
version "3.2.0"
resolved "https://registry.yarnpkg.com/fetch-blob/-/fetch-blob-3.2.0.tgz#f09b8d4bbd45adc6f0c20b7e787e793e309dcce9"
integrity sha512-7yAQpD2UMJzLi1Dqv7qFYnPbaPx7ZfFK6PiIxQ4PfkGPyNyl2Ugx+a/umUonmKqjhM4DnfbMvdX6otXq83soQQ==
dependencies:
node-domexception "^1.0.0"
web-streams-polyfill "^3.0.3"
fill-range@^7.0.1:
version "7.0.1"
resolved "https://registry.npmjs.org/fill-range/-/fill-range-7.0.1.tgz"
integrity sha512-qOo9F+dMUmC2Lcb4BbVvnKJxTPjCm+RRpe4gDuGrzkL7mEVl/djYSu2OdQ2Pa302N4oqkSg9ir6jaLWJ2USVpQ==
dependencies:
to-regex-range "^5.0.1"
form-data@^3.0.0:
version "3.0.1"
resolved "https://registry.yarnpkg.com/form-data/-/form-data-3.0.1.tgz#ebd53791b78356a99af9a300d4282c4d5eb9755f"
integrity sha512-RHkBKtLWUVwd7SqRIvCZMEvAMoGUp0XU+seQiZejj0COz3RI3hWP4sCv3gZWWLjJTd7rGwcsF5eKZGii0r/hbg==
dependencies:
asynckit "^0.4.0"
combined-stream "^1.0.8"
mime-types "^2.1.12"
formdata-polyfill@^4.0.10:
version "4.0.10"
resolved "https://registry.yarnpkg.com/formdata-polyfill/-/formdata-polyfill-4.0.10.tgz#24807c31c9d402e002ab3d8c720144ceb8848423"
integrity sha512-buewHzMvYL29jdeQTVILecSaZKnt/RJWjoZCF5OW60Z67/GmSLBkOFM7qh1PI3zFNtJbaZL5eQu1vLfazOwj4g==
dependencies:
fetch-blob "^3.1.2"
functional-red-black-tree@^1.0.1:
version "1.0.1"
resolved "https://registry.npmjs.org/functional-red-black-tree/-/functional-red-black-tree-1.0.1.tgz"
integrity sha512-dsKNQNdj6xA3T+QlADDA7mOSlX0qiMINjn0cgr+eGHGsbSHzTabcIogz2+p/iqP1Xs6EP/sS2SbqH+brGTbq0g==
glob-parent@^5.1.2:
version "5.1.2"
resolved "https://registry.npmjs.org/glob-parent/-/glob-parent-5.1.2.tgz"
integrity sha512-AOIgSQCepiJYwP3ARnGx+5VnTu2HBYdzbGP45eLw1vr3zB3vZLeyed1sC9hnbcOc9/SrMyM5RPQrkGz4aS9Zow==
dependencies:
is-glob "^4.0.1"
globby@^11.1.0:
version "11.1.0"
resolved "https://registry.npmjs.org/globby/-/globby-11.1.0.tgz"
integrity sha512-jhIXaOzy1sb8IyocaruWSn1TjmnBVs8Ayhcy83rmxNJ8q2uWKCAj3CnJY+KpGSXCueAPc0i05kVvVKtP1t9S3g==
dependencies:
array-union "^2.1.0"
dir-glob "^3.0.1"
fast-glob "^3.2.9"
ignore "^5.2.0"
merge2 "^1.4.1"
slash "^3.0.0"
ignore@^5.2.0:
version "5.2.4"
resolved "https://registry.npmjs.org/ignore/-/ignore-5.2.4.tgz"
integrity sha512-MAb38BcSbH0eHNBxn7ql2NH/kX33OkB3lZ1BNdh7ENeRChHTYsTvWrMubiIAMNS2llXEEgZ1MUOBtXChP3kaFQ==
is-extglob@^2.1.1:
version "2.1.1"
resolved "https://registry.npmjs.org/is-extglob/-/is-extglob-2.1.1.tgz"
integrity sha512-SbKbANkN603Vi4jEZv49LeVJMn4yGwsbzZworEoyEiutsN3nJYdbO36zfhGJ6QEDpOZIFkDtnq5JRxmvl3jsoQ==
is-glob@^4.0.1, is-glob@^4.0.3:
version "4.0.3"
resolved "https://registry.npmjs.org/is-glob/-/is-glob-4.0.3.tgz"
integrity sha512-xelSayHH36ZgE7ZWhli7pW34hNbNl8Ojv5KVmkJD4hBdD3th8Tfk9vYasLM+mXWOZhFkgZfxhLSnrwRr4elSSg==
dependencies:
is-extglob "^2.1.1"
is-number@^7.0.0:
version "7.0.0"
resolved "https://registry.npmjs.org/is-number/-/is-number-7.0.0.tgz"
integrity sha512-41Cifkg6e8TylSpdtTpeLVMqvSBEVzTttHvERD741+pnZ8ANv0004MRL43QKPDlK9cGvNp6NZWZUBlbGXYxxng==
lru-cache@^6.0.0:
version "6.0.0"
resolved "https://registry.npmjs.org/lru-cache/-/lru-cache-6.0.0.tgz"
integrity sha512-Jo6dJ04CmSjuznwJSS3pUeWmd/H0ffTlkXXgwZi+eq1UCmqQwCh+eLsYOYCwY991i2Fah4h1BEMCx4qThGbsiA==
dependencies:
yallist "^4.0.0"
merge2@^1.3.0, merge2@^1.4.1:
version "1.4.1"
resolved "https://registry.npmjs.org/merge2/-/merge2-1.4.1.tgz"
integrity sha512-8q7VEgMJW4J8tcfVPy8g09NcQwZdbwFEqhe/WZkoIzjn/3TGDwtOCYtXGxA3O8tPzpczCCDgv+P2P5y00ZJOOg==
micromatch@^4.0.4:
version "4.0.5"
resolved "https://registry.npmjs.org/micromatch/-/micromatch-4.0.5.tgz"
integrity sha512-DMy+ERcEW2q8Z2Po+WNXuw3c5YaUSFjAO5GsJqfEl7UjvtIuFKO6ZrKvcItdy98dwFI2N1tg3zNIdKaQT+aNdA==
dependencies:
braces "^3.0.2"
picomatch "^2.3.1"
mime-db@1.52.0:
version "1.52.0"
resolved "https://registry.yarnpkg.com/mime-db/-/mime-db-1.52.0.tgz#bbabcdc02859f4987301c856e3387ce5ec43bf70"
integrity sha512-sPU4uV7dYlvtWJxwwxHD0PuihVNiE7TyAbQ5SWxDCB9mUYvOgroQOwYQQOKPJ8CIbE+1ETVlOoK1UC2nU3gYvg==
mime-types@^2.1.12:
version "2.1.35"
resolved "https://registry.yarnpkg.com/mime-types/-/mime-types-2.1.35.tgz#381a871b62a734450660ae3deee44813f70d959a"
integrity sha512-ZDY+bPm5zTTF+YpCrAU9nK0UgICYPT0QtT1NZWFv4s++TNkcgVaT0g6+4R2uI4MjQjzysHB1zxuWL50hzaeXiw==
dependencies:
mime-db "1.52.0"
moment@2.29.4:
version "2.29.4"
resolved "https://registry.npmjs.org/moment/-/moment-2.29.4.tgz"
integrity sha512-5LC9SOxjSc2HF6vO2CyuTDNivEdoz2IvyJJGj6X8DJ0eFyfszE0QiEd+iXmBvUP3WHxSjFH/vIsA0EN00cgr8w==
ms@2.1.2:
version "2.1.2"
resolved "https://registry.npmjs.org/ms/-/ms-2.1.2.tgz"
integrity sha512-sGkPx+VjMtmA6MX27oA4FBFELFCZZ4S4XqeGOXCv68tT+jb3vk/RyaKWP0PTKyWtmLSM0b+adUTEvbs1PEaH2w==
node-domexception@^1.0.0:
version "1.0.0"
resolved "https://registry.yarnpkg.com/node-domexception/-/node-domexception-1.0.0.tgz#6888db46a1f71c0b76b3f7555016b63fe64766e5"
integrity sha512-/jKZoMpw0F8GRwl4/eLROPA3cfcXtLApP0QzLmUT/HuPCZWyB7IY9ZrMeKw2O/nFIqPQB3PVM9aYm0F312AXDQ==
node-fetch@^3.1.0:
version "3.3.1"
resolved "https://registry.yarnpkg.com/node-fetch/-/node-fetch-3.3.1.tgz#b3eea7b54b3a48020e46f4f88b9c5a7430d20b2e"
integrity sha512-cRVc/kyto/7E5shrWca1Wsea4y6tL9iYJE5FBCius3JQfb/4P4I295PfhgbJQBLTx6lATE4z+wK0rPM4VS2uow==
dependencies:
data-uri-to-buffer "^4.0.0"
fetch-blob "^3.1.4"
formdata-polyfill "^4.0.10"
obsidian@latest:
version "1.1.1"
resolved "https://registry.npmjs.org/obsidian/-/obsidian-1.1.1.tgz"
integrity sha512-GcxhsHNkPEkwHEjeyitfYNBcQuYGeAHFs1pEpZIv0CnzSfui8p8bPLm2YKLgcg20B764770B1sYGtxCvk9ptxg==
dependencies:
"@types/codemirror" "0.0.108"
moment "2.29.4"
path-type@^4.0.0:
version "4.0.0"
resolved "https://registry.npmjs.org/path-type/-/path-type-4.0.0.tgz"
integrity sha512-gDKb8aZMDeD/tZWs9P6+q0J9Mwkdl6xMV8TjnGP3qJVJ06bdMgkbBlLU8IdfOsIsFz2BW1rNVT3XuNEl8zPAvw==
picomatch@^2.3.1:
version "2.3.1"
resolved "https://registry.npmjs.org/picomatch/-/picomatch-2.3.1.tgz"
integrity sha512-JU3teHTNjmE2VCGFzuY8EXzCDVwEqB2a8fsIvwaStHhAWJEeVd1o1QD80CU6+ZdEXXSLbSsuLwJjkCBWqRQUVA==
queue-microtask@^1.2.2:
version "1.2.3"
resolved "https://registry.npmjs.org/queue-microtask/-/queue-microtask-1.2.3.tgz"
integrity sha512-NuaNSa6flKT5JaSYQzJok04JzTL1CA6aGhv5rfLW3PgqA+M2ChpZQnAC8h8i4ZFkBS8X5RqkDBHA7r4hej3K9A==
regexpp@^3.2.0:
version "3.2.0"
resolved "https://registry.npmjs.org/regexpp/-/regexpp-3.2.0.tgz"
integrity sha512-pq2bWo9mVD43nbts2wGv17XLiNLya+GklZ8kaDLV2Z08gDCsGpnKn9BFMepvWuHCbyVvY7J5o5+BVvoQbmlJLg==
reusify@^1.0.4:
version "1.0.4"
resolved "https://registry.npmjs.org/reusify/-/reusify-1.0.4.tgz"
integrity sha512-U9nH88a3fc/ekCF1l0/UP1IosiuIjyTh7hBvXVMHYgVcfGvt897Xguj2UOLDeI5BG2m7/uwyaLVT6fbtCwTyzw==
run-parallel@^1.1.9:
version "1.2.0"
resolved "https://registry.npmjs.org/run-parallel/-/run-parallel-1.2.0.tgz"
integrity sha512-5l4VyZR86LZ/lDxZTR6jqL8AFE2S0IFLMP26AbjsLVADxHdhB/c0GUsH+y39UfCi3dzz8OlQuPmnaJOMoDHQBA==
dependencies:
queue-microtask "^1.2.2"
semver@^7.3.7:
version "7.3.8"
resolved "https://registry.npmjs.org/semver/-/semver-7.3.8.tgz"
integrity sha512-NB1ctGL5rlHrPJtFDVIVzTyQylMLu9N9VICA6HSFJo8MCGVTMW6gfpicwKmmK/dAjTOrqu5l63JJOpDSrAis3A==
dependencies:
lru-cache "^6.0.0"
slash@^3.0.0:
version "3.0.0"
resolved "https://registry.npmjs.org/slash/-/slash-3.0.0.tgz"
integrity sha512-g9Q1haeby36OSStwb4ntCGGGaKsaVSjQ68fBxoQcutl5fS1vuY18H3wSt3jFyFtrkx+Kz0V1G85A4MyAdDMi2Q==
to-regex-range@^5.0.1:
version "5.0.1"
resolved "https://registry.npmjs.org/to-regex-range/-/to-regex-range-5.0.1.tgz"
integrity sha512-65P7iz6X5yEr1cwcgvQxbbIw7Uk3gOy5dIdtZ4rDveLqhrdJP+Li/Hx6tyK0NEb+2GCyneCMJiGqrADCSNk8sQ==
dependencies:
is-number "^7.0.0"
tslib@2.4.0:
version "2.4.0"
resolved "https://registry.npmjs.org/tslib/-/tslib-2.4.0.tgz"
integrity sha512-d6xOpEDfsi2CZVlPQzGeux8XMwLT9hssAsaPYExaQMuYskwb+x1x7J371tWlbBdWHroy99KnVB6qIkUbs5X3UQ==
tslib@^1.8.1:
version "1.14.1"
resolved "https://registry.npmjs.org/tslib/-/tslib-1.14.1.tgz"
integrity sha512-Xni35NKzjgMrwevysHTCArtLDpPvye8zV/0E4EyYn43P7/7qvQwPh9BGkHewbMulVntbigmcT7rdX3BNo9wRJg==
tsutils@^3.21.0:
version "3.21.0"
resolved "https://registry.npmjs.org/tsutils/-/tsutils-3.21.0.tgz"
integrity sha512-mHKK3iUXL+3UF6xL5k0PEhKRUBKPBCv/+RkEOpjRWxxx27KKRBmmA60A9pgOUvMi8GKhRMPEmjBRPzs2W7O1OA==
dependencies:
tslib "^1.8.1"
typescript@4.7.4:
version "4.7.4"
resolved "https://registry.yarnpkg.com/typescript/-/typescript-4.7.4.tgz#1a88596d1cf47d59507a1bcdfb5b9dfe4d488235"
integrity sha512-C0WQT0gezHuw6AdY1M2jxUO83Rjf0HP7Sk1DtXj6j1EwkQNZrHAg2XPWlq62oqEhYvONq5pkC2Y9oPljWToLmQ==
web-streams-polyfill@^3.0.3:
version "3.2.1"
resolved "https://registry.yarnpkg.com/web-streams-polyfill/-/web-streams-polyfill-3.2.1.tgz#71c2718c52b45fd49dbeee88634b3a60ceab42a6"
integrity sha512-e0MO3wdXWKrLbL0DgGnUV7WHVuw9OUvL4hjgnPkIeEvESk74gAITi5G606JtZPp39cd8HA9VQzCIvA49LpPN5Q==
yallist@^4.0.0:
version "4.0.0"
resolved "https://registry.npmjs.org/yallist/-/yallist-4.0.0.tgz"
integrity sha512-3wdGidZyq5PB084XLES5TpOSRA3wjXAlIWMhum2kRcv/41Sn2emQ0dycQW4uZXLejwKvg6EsvbdlVL+FYEct7A==

29
src/interface/web/assets/config.css
View File

@@ -1,29 +0,0 @@
:root {
--primary-color: #ffffff;
--bold-color: #2073ee;
--complementary-color: #124408;
--accent-color-0: #57f0b5;
}
input[type=text] {
width: 40%;
}
div.config-element {
color: var(--bold-color);
margin: 8px;
}
div.config-title {
font-weight: bold;
}
span.config-element-value {
color: var(--complementary-color);
font-weight: normal;
cursor: pointer;
}
button {
cursor: pointer;
}

125
src/interface/web/assets/config.js
View File

@@ -1,125 +0,0 @@
// Retrieve elements from the DOM.
var showConfig = document.getElementById("show-config");
var configForm = document.getElementById("config-form");
var regenerateButton = document.getElementById("config-regenerate");
// Global variables.
var rawConfig = {};
var emptyValueDefault = "🖊️";
/**
* Fetch the existing config file.
*/
fetch("/api/config/data")
.then(response => response.json())
.then(data => {
rawConfig = data;
configForm.style.display = "block";
processChildren(configForm, data);
var submitButton = document.createElement("button");
submitButton.type = "submit";
submitButton.innerHTML = "update";
configForm.appendChild(submitButton);
// The config form's submit handler.
configForm.addEventListener("submit", (event) => {
event.preventDefault();
console.log(rawConfig);
fetch("/api/config/data", {
method: "POST",
credentials: "same-origin",
headers: {
'Content-Type': 'application/json'
},
body: JSON.stringify(rawConfig)
})
.then(response => response.json())
.then(data => console.log(data));
});
});
/**
* The click handler for the Regenerate button.
*/
regenerateButton.addEventListener("click", (event) => {
event.preventDefault();
regenerateButton.style.cursor = "progress";
regenerateButton.disabled = true;
fetch("/api/update?force=true")
.then(response => response.json())
.then(data => {
regenerateButton.style.cursor = "pointer";
regenerateButton.disabled = false;
console.log(data);
});
})
/**
* Adds config elements to the DOM representing the sub-components
* of one of the fields in the raw config file.
* @param {the parent element} element
* @param {the data to be rendered for this element and its children} data
*/
function processChildren(element, data) {
for (let key in data) {
var child = document.createElement("div");
child.id = key;
child.className = "config-element";
child.appendChild(document.createTextNode(key + ": "));
if (data[key] === Object(data[key]) && !Array.isArray(data[key])) {
child.className+=" config-title";
processChildren(child, data[key]);
} else {
child.appendChild(createValueNode(data, key));
}
element.appendChild(child);
}
}
/**
* Takes an element, and replaces it with an editable
* element with the same data in place.
* @param {the original element to be replaced} original
* @param {the source data to be rendered for the new element} data
* @param {the key for this input in the source data} key
*/
function makeElementEditable(original, data, key) {
original.addEventListener("click", () => {
var inputNewText = document.createElement("input");
inputNewText.type = "text";
inputNewText.className = "config-element-edit";
inputNewText.value = (original.textContent == emptyValueDefault) ? "" : original.textContent;
fixInputOnFocusOut(inputNewText, data, key);
original.parentNode.replaceChild(inputNewText, original);
inputNewText.focus();
});
}
/**
* Creates a node corresponding to the value of a config element.
* @param {the source data} data
* @param {the key corresponding to this node's data} key
* @returns A new element which corresponds to the value in some field.
*/
function createValueNode(data, key) {
var valueElement = document.createElement("span");
valueElement.className = "config-element-value";
valueElement.textContent = !data[key] ? emptyValueDefault : data[key];
makeElementEditable(valueElement, data, key);
return valueElement;
}
/**
* Replaces an existing input element with an element with the same data, which is not an input.
* If the input data for this element was changed, update the corresponding data in the raw config.
* @param {the original element to be replaced} original
* @param {the source data} data
* @param {the key corresponding to this node's data} key
*/
function fixInputOnFocusOut(original, data, key) {
original.addEventListener("blur", () => {
data[key] = (original.value != emptyValueDefault) ? original.value : "";
original.parentNode.replaceChild(createValueNode(data, key), original);
})
}

BIN
src/interface/web/assets/icons/favicon-128x128.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 26 KiB

BIN
src/interface/web/assets/icons/favicon-144x144.ico
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 159 KiB

BIN
src/interface/web/assets/icons/favicon-144x144.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 29 KiB

BIN
src/interface/web/assets/icons/favicon.icns
View File

Binary file not shown.

261
src/interface/web/chat.html
View File

@@ -1,261 +0,0 @@
<html>
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0 maximum-scale=1.0">
<title>Khoj</title>
<link rel="icon" href="data:image/svg+xml,<svg xmlns=%22http://www.w3.org/2000/svg%22 viewBox=%220 0 144 144%22><text y=%22.86em%22 font-size=%22144%22>🦅</text></svg>">
<link rel="icon" type="image/png" sizes="144x144" href="/static/assets/icons/favicon-144x144.png">
<link rel="manifest" href="/static/khoj.webmanifest">
</head>
<script>
function setTypeFieldInUrl(type) {
let url = new URL(window.location.href);
url.searchParams.set("t", type.value);
window.history.pushState({}, "", url.href);
}
function formatDate(date) {
// Format date in HH:MM, DD MMM YYYY format
let time_string = date.toLocaleTimeString('en-IN', { hour: '2-digit', minute: '2-digit', hour12: false });
let date_string = date.toLocaleString('en-IN', { year: 'numeric', month: 'short', day: '2-digit'}).replaceAll('-', ' ');
return `${time_string}, ${date_string}`;
}
function renderMessage(message, by, dt=null) {
let message_time = formatDate(dt ?? new Date());
let by_name = by == "khoj" ? "🦅 Khoj" : "🤔 You";
// Generate HTML for Chat Message and Append to Chat Body
document.getElementById("chat-body").innerHTML += `
<div data-meta="${by_name} at ${message_time}" class="chat-message ${by}">
<div class="chat-message-text ${by}">${message}</div>
</div>
`;
// Scroll to bottom of input-body element
document.getElementById("chat-body").scrollTop = document.getElementById("chat-body").scrollHeight;
}
function chat() {
// Extract required fields for search from form
query = document.getElementById("chat-input").value.trim();
type_ = document.getElementById("chat-type").value;
console.log(`Query: ${query}, Type: ${type_}`);
// Short circuit on empty query
if (query.length === 0)
return;
// Add message by user to chat body
renderMessage(query, "you");
document.getElementById("chat-input").value = "";
// Generate backend API URL to execute query
url = type_ === "chat"
? `/api/beta/chat?q=${encodeURIComponent(query)}`
: `/api/beta/summarize?q=${encodeURIComponent(query)}`;
// Call specified Khoj API
fetch(url)
.then(response => response.json())
.then(data => data.response)
.then(response => {
// Render message by Khoj to chat body
console.log(response);
renderMessage(response, "khoj");
});
}
function incrementalChat(event) {
// Send chat message on 'Enter'
if (event.key === 'Enter') {
chat();
}
}
window.onload = function () {
// Fill type field with value passed in URL query parameters, if any.
var type_via_url = new URLSearchParams(window.location.search).get("t");
if (type_via_url)
document.getElementById("chat-type").value = type_via_url;
fetch('/api/beta/chat')
.then(response => response.json())
.then(data => data.response)
.then(chat_logs => {
// Render conversation history, if any
chat_logs.forEach(chat_log => {
renderMessage(chat_log.message, chat_log.by, new Date(chat_log.created));
});
});
// Set welcome message on load
renderMessage("Hey, what's up?", "khoj");
// Fill query field with value passed in URL query parameters, if any.
var query_via_url = new URLSearchParams(window.location.search).get("q");
if (query_via_url) {
document.getElementById("chat-input").value = query_via_url;
chat();
}
}
</script>
<body>
<!-- Chat Header -->
<h1>Khoj</h1>
<!-- Chat Body -->
<div id="chat-body"></div>
<!-- Chat Footer -->
<div id="chat-footer">
<input type="text" id="chat-input" class="option" onkeyup=incrementalChat(event) autofocus="autofocus" placeholder="What is the meaning of life?">
<!--Select Chat Type from: Chat, Summarize -->
<select id="chat-type" class="option" onchange="setTypeFieldInUrl(this)">
<option value="chat">Chat</option>
<option value="summarize">Summarize</option>
</select>
</div>
</body>
<style>
html, body {
height: 100%;
width: 100%;
padding: 0px;
margin: 0px;
}
body {
display: grid;
background: #f8fafc;
color: #475569;
text-align: center;
font-family: roboto, karma, segoe ui, sans-serif;
font-size: 20px;
font-weight: 300;
line-height: 1.5em;
}
body > * {
padding: 10px;
margin: 10px;
}
h1 {
font-weight: 200;
color: #017eff;
}
#chat-body {
font-size: medium;
margin: 0px;
line-height: 20px;
overflow-y: scroll; /* Make chat body scroll to see history */
}
/* add chat metatdata to bottom of bubble */
.chat-message::after {
content: attr(data-meta);
display: block;
font-size: x-small;
color: #475569;
margin: -12px 7px 0 -5px;
}
/* move message by khoj to left */
.chat-message.khoj {
margin-left: auto;
text-align: left;
}
/* move message by you to right */
.chat-message.you {
margin-right: auto;
text-align: right;
}
/* basic style chat message text */
.chat-message-text {
margin: 10px;
border-radius: 10px;
padding: 10px;
position: relative;
display: inline-block;
max-width: 80%;
text-align: left;
}
/* color chat bubble by khoj blue */
.chat-message-text.khoj {
color: #f8fafc;
background: #017eff;
margin-left: auto;
}
/* add left protrusion to khoj chat bubble */
.chat-message-text.khoj:after {
content: '';
position: absolute;
bottom: -2px;
left: -7px;
border: 10px solid transparent;
border-top-color: #017eff;
border-bottom: 0;
transform: rotate(-60deg);
}
/* color chat bubble by you dark grey */
.chat-message-text.you {
color: #f8fafc;
background: #475569;
margin-right: auto;
}
/* add right protrusion to you chat bubble */
.chat-message-text.you:after {
content: '';
position: absolute;
top: 91%;
right: -2px;
border: 10px solid transparent;
border-left-color: #475569;
border-right: 0;
margin-top: -10px;
transform: rotate(-60deg)
}
#chat-footer {
padding: 0;
display: grid;
grid-template-columns: minmax(70px, 85%) auto;
grid-column-gap: 10px;
grid-row-gap: 10px;
}
#chat-footer > * {
padding: 15px;
border-radius: 5px;
border: 1px solid #475569;
background: #f9fafc
}
.option:hover {
box-shadow: 0 0 11px #aaa;
}
#chat-input {
font-size: medium;
}
@media only screen and (max-width: 600px) {
body {
grid-template-columns: 1fr;
grid-template-rows: auto minmax(80px, 100%) auto;
}
body > * {
grid-column: 1;
}
#chat-footer {
padding: 0;
margin: 4px;
grid-template-columns: auto;
}
}
@media only screen and (min-width: 600px) {
body {
grid-template-columns: auto min(70vw, 100%) auto;
grid-template-rows: auto minmax(80px, 100%) auto;
}
body > * {
grid-column: 2;
}
}
</style>
</html>

14
src/interface/web/config.html
View File

@@ -1,14 +0,0 @@
<!DOCTYPE html>
<head>
<meta charset="utf-8">
<link rel="icon" href="data:image/svg+xml,<svg xmlns=%22http://www.w3.org/2000/svg%22 viewBox=%220 0 100 100%22><text y=%22.9em%22 font-size=%2290%22>🦅</text></svg>">
<link rel="stylesheet" href="static/assets/config.css">
<title>Khoj - Configure App</title>
</head>
<body>
<form id="config-form">
</form>
<button id="config-regenerate">regenerate</button>
</body>
<script src="static/assets/config.js"></script>
</html>

326
src/interface/web/index.html
View File

@@ -1,326 +0,0 @@
<html>
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0 maximum-scale=1.0">
<title>Khoj</title>
<link rel="icon" href="data:image/svg+xml,<svg xmlns=%22http://www.w3.org/2000/svg%22 viewBox=%220 0 144 144%22><text y=%22.86em%22 font-size=%22144%22>🦅</text></svg>">
<link rel="icon" type="image/png" sizes="144x144" href="/static/assets/icons/favicon-144x144.png">
<link rel="manifest" href="/static/khoj.webmanifest">
</head>
<script type="text/javascript" src="static/assets/org.min.js"></script>
<script type="text/javascript" src="static/assets/markdown-it.min.js"></script>
<script>
function render_image(item) {
return `
<a href="${item.entry}" class="image-link">
<img id=${item.score} src="${item.entry}?${Math.random()}"
title="Effective Score: ${item.score}, Meta: ${item.additional.metadata_score}, Image: ${item.additional.image_score}"
class="image">
</a>`
}
function render_org(query, data, classPrefix="") {
var orgCode = data.map(function (item) {
return `${item.entry}`
}).join("\n")
var orgParser = new Org.Parser();
var orgDocument = orgParser.parse(orgCode);
var orgHTMLDocument = orgDocument.convert(Org.ConverterHTML, { htmlClassPrefix: classPrefix });
return orgHTMLDocument.toString();
}
function render_markdown(query, data) {
var md = window.markdownit();
return md.render(data.map(function (item) {
return `${item.entry}`
}).join("\n"));
}
function render_ledger(query, data) {
return `<div id="results-ledger">` + data.map(function (item) {
return `<p>${item.entry}</p>`
}).join("\n") + `</div>`;
}
function render_json(data, query, type) {
if (type === "markdown") {
return render_markdown(query, data);
} else if (type === "org") {
return render_org(query, data);
} else if (type === "music") {
return render_org(query, data, "music-");
} else if (type === "image") {
return data.map(render_image).join('');
} else if (type === "ledger") {
return render_ledger(query, data);
} else {
return `<pre id="json">${JSON.stringify(data, null, 2)}</pre>`;
}
}
function search(rerank=false) {
// Extract required fields for search from form
query = document.getElementById("query").value.trim();
type = document.getElementById("type").value;
results_count = document.getElementById("results-count").value || 6;
console.log(`Query: ${query}, Type: ${type}`);
// Short circuit on empty query
if (query.length === 0)
return;
// If set query field in url query param on rerank
if (rerank)
setQueryFieldInUrl(query);
// Generate Backend API URL to execute Search
url = type === "image"
? `/api/search?q=${encodeURIComponent(query)}&t=${type}&n=${results_count}`
: `/api/search?q=${encodeURIComponent(query)}&t=${type}&n=${results_count}&r=${rerank}`;
// Execute Search and Render Results
fetch(url)
.then(response => response.json())
.then(data => {
console.log(data);
document.getElementById("results").innerHTML =
`<div id=results-${type}>`
+ render_json(data, query, type)
+ `</div>`;
});
}
function updateIndex() {
type = document.getElementById("type").value;
fetch(`/api/update?t=${type}`)
.then(response => response.json())
.then(data => {
console.log(data);
document.getElementById("results").innerHTML =
render_json(data);
});
}
function incrementalSearch(event) {
type = document.getElementById("type").value;
// Search with reranking on 'Enter'
if (event.key === 'Enter') {
search(rerank=true);
}
// Limit incremental search to text types
else if (type !== "image") {
search(rerank=false);
}
}
function populate_type_dropdown() {
// Populate type dropdown field with enabled search types only
var possible_search_types = ["org", "markdown", "ledger", "music", "image"];
fetch("/api/config/data")
.then(response => response.json())
.then(data => {
document.getElementById("type").innerHTML =
possible_search_types
.filter(type => data["content-type"].hasOwnProperty(type) && data["content-type"][type])
.map(type => `<option value="${type}">${type.slice(0,1).toUpperCase() + type.slice(1)}</option>`)
.join('');
})
.then(() => {
// Set type field to search type passed in URL query parameter, if valid
var type_via_url = new URLSearchParams(window.location.search).get("t");
if (type_via_url && possible_search_types.includes(type_via_url))
document.getElementById("type").value = type_via_url;
});
}
function setTypeFieldInUrl(type) {
var url = new URL(window.location.href);
url.searchParams.set("t", type.value);
window.history.pushState({}, "", url.href);
}
function setCountFieldInUrl(results_count) {
var url = new URL(window.location.href);
url.searchParams.set("n", results_count.value);
window.history.pushState({}, "", url.href);
}
function setQueryFieldInUrl(query) {
var url = new URL(window.location.href);
url.searchParams.set("q", query);
window.history.pushState({}, "", url.href);
}
window.onload = function () {
// Dynamically populate type dropdown based on enabled search types and type passed as URL query parameter
populate_type_dropdown();
// Set results count field with value passed in URL query parameters, if any.
var results_count = new URLSearchParams(window.location.search).get("n");
if (results_count)
document.getElementById("results-count").value = results_count;
// Fill query field with value passed in URL query parameters, if any.
var query_via_url = new URLSearchParams(window.location.search).get("q");
if (query_via_url)
document.getElementById("query").value = query_via_url;
}
</script>
<body>
<h1>Khoj</h1>
<!--Add Text Box To Enter Query, Trigger Incremental Search OnChange -->
<input type="text" id="query" class="option" onkeyup=incrementalSearch(event) autofocus="autofocus" placeholder="What is the meaning of life?">
<div id="options">
<!--Add Dropdown to Select Query Type -->
<select id="type" class="option" onchange="setTypeFieldInUrl(this)"></select>
<!--Add Button To Regenerate -->
<button id="update" class="option" onclick="updateIndex()">Update</button>
<!--Add Results Count Input To Set Results Count -->
<input type="number" id="results-count" min="1" max="100" value="6" placeholder="results count" onchange="setCountFieldInUrl(this)">
</div>
<!-- Section to Render Results -->
<div id="results"></div>
</body>
<style>
@media only screen and (max-width: 600px) {
body {
display: grid;
grid-template-columns: 1fr;
grid-template-rows: 1fr 1fr 1fr minmax(80px, 100%);
}
body > * {
grid-column: 1;
}
}
@media only screen and (min-width: 600px) {
body {
display: grid;
grid-template-columns: 1fr min(70vw, 100%) 1fr;
grid-template-rows: 1fr 1fr 1fr minmax(80px, 100%);
padding-top: 60vw;
}
body > * {
grid-column: 2;
}
}
body {
padding: 0px;
margin: 0px;
background: #f8fafc;
color: #475569;
text-align: center;
font-family: roboto, karma, segoe ui, sans-serif;
font-size: 20px;
font-weight: 300;
line-height: 1.5em;
}
body > * {
padding: 10px;
margin: 10px;
}
h1 {
font-weight: 200;
color: #017eff;
}
#options {
padding: 0;
display: grid;
grid-template-columns: 1fr 1fr minmax(70px, 0.5fr);
}
#options > * {
padding: 15px;
border-radius: 5px;
border: 1px solid #475569;
background: #f9fafc
}
.option:hover {
box-shadow: 0 0 11px #aaa;
}
#options > select {
margin-right: 10px;
}
#options > button {
margin-right: 10px;
}
#query {
font-size: larger;
}
#results {
font-size: medium;
margin: 0px;
line-height: 20px;
}
#results-image {
display: grid;
grid-template-columns: repeat(3, 1fr);
}
.image-link {
place-self: center;
}
.image {
width: 20vw;
border-radius: 10px;
border: 1px solid #475569;
}
#json {
white-space: pre-wrap;
}
#results-ledger {
white-space: pre-line;
text-align: left;
}
#results-markdown {
text-align: left;
}
#results-music,
#results-org {
text-align: left;
white-space: pre-line;
}
#results-music h3,
#results-org h3 {
margin: 20px 0 0 0;
font-size: larger;
}
span.music-task-status,
span.task-status {
color: white;
padding: 3.5px 3.5px 0;
margin-right: 5px;
border-radius: 5px;
background-color: #eab308;
font-size: medium;
}
span.music-task-status.todo,
span.task-status.todo {
background-color: #3b82f6
}
span.music-task-status.done,
span.task-status.done {
background-color: #22c55e;
}
span.music-task-tag,
span.task-tag {
color: white;
padding: 3.5px 3.5px 0;
margin-right: 5px;
border-radius: 5px;
border: 1px solid #475569;
background-color: #ef4444;
font-size: small;
}
</style>
</html>

0
src/__init__.py → src/khoj/__init__.py
View File

282
src/khoj/configure.py Normal file
View File

@@ -0,0 +1,282 @@
# Standard Packages
import sys
import logging
import json
from enum import Enum
from typing import Optional
import requests
# External Packages
import schedule
from fastapi.staticfiles import StaticFiles
# Internal Packages
from khoj.processor.conversation.gpt import summarize
from khoj.processor.jsonl.jsonl_to_jsonl import JsonlToJsonl
from khoj.processor.markdown.markdown_to_jsonl import MarkdownToJsonl
from khoj.processor.org_mode.org_to_jsonl import OrgToJsonl
from khoj.processor.pdf.pdf_to_jsonl import PdfToJsonl
from khoj.processor.github.github_to_jsonl import GithubToJsonl
from khoj.processor.notion.notion_to_jsonl import NotionToJsonl
from khoj.search_type import image_search, text_search
from khoj.utils import constants, state
from khoj.utils.config import SearchType, SearchModels, ProcessorConfigModel, ConversationProcessorConfigModel
from khoj.utils.helpers import LRU, resolve_absolute_path, merge_dicts
from khoj.utils.rawconfig import FullConfig, ProcessorConfig
from khoj.search_filter.date_filter import DateFilter
from khoj.search_filter.word_filter import WordFilter
from khoj.search_filter.file_filter import FileFilter
logger = logging.getLogger(__name__)
def configure_server(args, required=False):
if args.config is None:
if required:
logger.error(
f"Exiting as Khoj is not configured.\nConfigure it via http://localhost:8000/config or by editing {state.config_file}."
)
sys.exit(1)
else:
logger.warning(
f"Khoj is not configured.\nConfigure it via http://localhost:8000/config, plugins or by editing {state.config_file}."
)
return
else:
state.config = args.config
# Initialize Processor from Config
state.processor_config = configure_processor(args.config.processor)
# Initialize the search type and model from Config
state.search_index_lock.acquire()
state.SearchType = configure_search_types(state.config)
state.model = configure_search(state.model, state.config, args.regenerate)
state.search_index_lock.release()
def configure_routes(app):
# Import APIs here to setup search types before while configuring server
from khoj.routers.api import api
from khoj.routers.api_beta import api_beta
from khoj.routers.web_client import web_client
app.mount("/static", StaticFiles(directory=constants.web_directory), name="static")
app.include_router(api, prefix="/api")
app.include_router(api_beta, prefix="/api/beta")
app.include_router(web_client)
if not state.demo:
@schedule.repeat(schedule.every(61).minutes)
def update_search_index():
state.search_index_lock.acquire()
state.model = configure_search(state.model, state.config, regenerate=False)
state.search_index_lock.release()
logger.info("📬 Search index updated via Scheduler")
def configure_search_types(config: FullConfig):
# Extract core search types
core_search_types = {e.name: e.value for e in SearchType}
# Extract configured plugin search types
plugin_search_types = {}
if config.content_type and config.content_type.plugins:
plugin_search_types = {plugin_type: plugin_type for plugin_type in config.content_type.plugins.keys()}
# Dynamically generate search type enum by merging core search types with configured plugin search types
return Enum("SearchType", merge_dicts(core_search_types, plugin_search_types))
def configure_search(model: SearchModels, config: FullConfig, regenerate: bool, t: Optional[state.SearchType] = None):
if config is None or config.content_type is None or config.search_type is None:
logger.warning("🚨 No Content or Search type is configured.")
return
if model is None:
model = SearchModels()
try:
# Initialize Org Notes Search
if (t == state.SearchType.Org or t == None) and config.content_type.org and config.search_type.asymmetric:
logger.info("🦄 Setting up search for orgmode notes")
# Extract Entries, Generate Notes Embeddings
model.org_search = text_search.setup(
OrgToJsonl,
config.content_type.org,
search_config=config.search_type.asymmetric,
regenerate=regenerate,
filters=[DateFilter(), WordFilter(), FileFilter()],
)
# Initialize Markdown Search
if (
(t == state.SearchType.Markdown or t == None)
and config.content_type.markdown
and config.search_type.asymmetric
):
logger.info("💎 Setting up search for markdown notes")
# Extract Entries, Generate Markdown Embeddings
model.markdown_search = text_search.setup(
MarkdownToJsonl,
config.content_type.markdown,
search_config=config.search_type.asymmetric,
regenerate=regenerate,
filters=[DateFilter(), WordFilter(), FileFilter()],
)
# Initialize PDF Search
if (t == state.SearchType.Pdf or t == None) and config.content_type.pdf and config.search_type.asymmetric:
logger.info("🖨️ Setting up search for pdf")
# Extract Entries, Generate PDF Embeddings
model.pdf_search = text_search.setup(
PdfToJsonl,
config.content_type.pdf,
search_config=config.search_type.asymmetric,
regenerate=regenerate,
filters=[DateFilter(), WordFilter(), FileFilter()],
)
# Initialize Image Search
if (t == state.SearchType.Image or t == None) and config.content_type.image and config.search_type.image:
logger.info("🌄 Setting up search for images")
# Extract Entries, Generate Image Embeddings
model.image_search = image_search.setup(
config.content_type.image, search_config=config.search_type.image, regenerate=regenerate
)
if (t == state.SearchType.Github or t == None) and config.content_type.github and config.search_type.asymmetric:
logger.info("🐙 Setting up search for github")
# Extract Entries, Generate Github Embeddings
model.github_search = text_search.setup(
GithubToJsonl,
config.content_type.github,
search_config=config.search_type.asymmetric,
regenerate=regenerate,
filters=[DateFilter(), WordFilter(), FileFilter()],
)
# Initialize External Plugin Search
if (t == None or t in state.SearchType) and config.content_type.plugins:
logger.info("🔌 Setting up search for plugins")
model.plugin_search = {}
for plugin_type, plugin_config in config.content_type.plugins.items():
model.plugin_search[plugin_type] = text_search.setup(
JsonlToJsonl,
plugin_config,
search_config=config.search_type.asymmetric,
regenerate=regenerate,
filters=[DateFilter(), WordFilter(), FileFilter()],
)
# Initialize Notion Search
if (t == None or t in state.SearchType) and config.content_type.notion:
logger.info("🔌 Setting up search for notion")
model.notion_search = text_search.setup(
NotionToJsonl,
config.content_type.notion,
search_config=config.search_type.asymmetric,
regenerate=regenerate,
filters=[DateFilter(), WordFilter(), FileFilter()],
)
except Exception as e:
logger.error("🚨 Failed to setup search")
raise e
# Invalidate Query Cache
state.query_cache = LRU()
return model
def configure_processor(processor_config: ProcessorConfig):
if not processor_config:
return
processor = ProcessorConfigModel()
# Initialize Conversation Processor
if processor_config.conversation:
logger.info("💬 Setting up conversation processor")
processor.conversation = configure_conversation_processor(processor_config.conversation)
return processor
def configure_conversation_processor(conversation_processor_config):
conversation_processor = ConversationProcessorConfigModel(conversation_processor_config)
conversation_logfile = resolve_absolute_path(conversation_processor.conversation_logfile)
if conversation_logfile.is_file():
# Load Metadata Logs from Conversation Logfile
with conversation_logfile.open("r") as f:
conversation_processor.meta_log = json.load(f)
logger.debug(f"Loaded conversation logs from {conversation_logfile}")
else:
# Initialize Conversation Logs
conversation_processor.meta_log = {}
conversation_processor.chat_session = []
return conversation_processor
@schedule.repeat(schedule.every(17).minutes)
def save_chat_session():
# No need to create empty log file
if not (
state.processor_config
and state.processor_config.conversation
and state.processor_config.conversation.meta_log
and state.processor_config.conversation.chat_session
):
return
# Summarize Conversation Logs for this Session
chat_session = state.processor_config.conversation.chat_session
openai_api_key = state.processor_config.conversation.openai_api_key
conversation_log = state.processor_config.conversation.meta_log
chat_model = state.processor_config.conversation.chat_model
session = {
"summary": summarize(chat_session, model=chat_model, api_key=openai_api_key),
"session-start": conversation_log.get("session", [{"session-end": 0}])[-1]["session-end"],
"session-end": len(conversation_log["chat"]),
}
if "session" in conversation_log:
conversation_log["session"].append(session)
else:
conversation_log["session"] = [session]
# Save Conversation Metadata Logs to Disk
conversation_logfile = resolve_absolute_path(state.processor_config.conversation.conversation_logfile)
conversation_logfile.parent.mkdir(parents=True, exist_ok=True) # create conversation directory if doesn't exist
with open(conversation_logfile, "w+", encoding="utf-8") as logfile:
json.dump(conversation_log, logfile, indent=2)
state.processor_config.conversation.chat_session = []
logger.info("📩 Saved current chat session to conversation logs")
@schedule.repeat(schedule.every(59).minutes)
def upload_telemetry():
if not state.config or not state.config.app or not state.config.app.should_log_telemetry or not state.telemetry:
message = "📡 No telemetry to upload" if not state.telemetry else "📡 Telemetry logging disabled"
logger.debug(message)
return
try:
logger.debug(f"📡 Upload usage telemetry to {constants.telemetry_server}:\n{state.telemetry}")
for log in state.telemetry:
for field in log:
# Check if the value for the field is JSON serializable
try:
json.dumps(log[field])
except TypeError:
log[field] = str(log[field])
requests.post(constants.telemetry_server, json=state.telemetry)
except Exception as e:
logger.error(f"📡 Error uploading telemetry: {e}")
else:
state.telemetry = []

0
src/interface/desktop/__init__.py → src/khoj/interface/desktop/__init__.py
View File

56
src/khoj/interface/desktop/main_window.py Normal file
View File

@@ -0,0 +1,56 @@
# Standard Packages
import webbrowser
# External Packages
from PyQt6 import QtGui, QtWidgets
from PyQt6.QtCore import Qt
# Internal Packages
from khoj.utils import constants
class MainWindow(QtWidgets.QMainWindow):
"""Create Window to Navigate users to the web UI"""
def __init__(self, host: str, port: int):
super(MainWindow, self).__init__()
# Initialize Configure Window
self.setWindowTitle("Khoj")
# Set Window Icon
icon_path = constants.web_directory / "assets/icons/favicon-128x128.png"
self.setWindowIcon(QtGui.QIcon(f"{icon_path.absolute()}"))
# Initialize Configure Window Layout
self.wlayout = QtWidgets.QVBoxLayout()
# Add a Label that says "Khoj Configuration" to the Window
self.wlayout.addWidget(QtWidgets.QLabel("Welcome to Khoj"))
# Add a Button to open the Web UI at http://host:port/config
self.open_web_ui_button = QtWidgets.QPushButton("Open Web UI")
self.open_web_ui_button.clicked.connect(lambda: webbrowser.open(f"http://{host}:{port}/config"))
self.wlayout.addWidget(self.open_web_ui_button)
# Set the central widget of the Window. Widget will expand
# to take up all the space in the window by default.
self.config_window = QtWidgets.QWidget()
self.config_window.setLayout(self.wlayout)
self.setCentralWidget(self.config_window)
self.position_window()
def position_window(self):
"Position the window at center of X axis and near top on Y axis"
window_rectangle = self.geometry()
screen_center = self.screen().availableGeometry().center()
window_rectangle.moveCenter(screen_center)
self.move(window_rectangle.topLeft().x(), 25)
def show_on_top(self):
"Bring Window on Top"
self.show()
self.setWindowState(Qt.WindowState.WindowActive)
self.activateWindow() # For Bringing to Top on Windows
self.raise_() # For Bringing to Top from Minimized State on OSX

17
src/interface/desktop/system_tray.py → src/khoj/interface/desktop/system_tray.py
View File

@@ -5,8 +5,8 @@ import webbrowser
from PyQt6 import QtGui, QtWidgets
# Internal Packages
from src.utils import constants, state
from src.interface.desktop.main_window import MainWindow
from khoj.utils import constants, state
from khoj.interface.desktop.main_window import MainWindow
def create_system_tray(gui: QtWidgets.QApplication, main_window: MainWindow):
@@ -17,23 +17,24 @@ def create_system_tray(gui: QtWidgets.QApplication, main_window: MainWindow):
"""
# Create the system tray with icon
icon_path = constants.web_directory / 'assets/icons/favicon-144x144.png'
icon = QtGui.QIcon(f'{icon_path.absolute()}')
icon_path = constants.web_directory / "assets/icons/favicon-128x128.png"
icon = QtGui.QIcon(f"{icon_path.absolute()}")
tray = QtWidgets.QSystemTrayIcon(icon)
tray.setVisible(True)
# Create the menu and menu actions
menu = QtWidgets.QMenu()
menu_actions = [
('Search', lambda: webbrowser.open(f'http://{state.host}:{state.port}/')),
('Configure', main_window.show_on_top),
('Quit', gui.quit),
("Search", lambda: webbrowser.open(f"http://{state.host}:{state.port}/")),
("Configure", lambda: webbrowser.open(f"http://{state.host}:{state.port}/config")),
("App", main_window.show),
("Quit", gui.quit),
]
# Add the menu actions to the menu
for action_text, action_function in menu_actions:
menu_action = QtGui.QAction(action_text, menu)
menu_action.triggered.connect(action_function)
menu_action.triggered.connect(action_function) # type: ignore[attr-defined]
menu.addAction(menu_action)
# Add the menu to the system tray

22
src/khoj/interface/web/404.html Normal file
View File

@@ -0,0 +1,22 @@
<!DOCTYPE html>
<html>
<head>
<title>Khoj: An AI Personal Assistant for your digital brain</title>
<link rel=”stylesheet” href=”static/styles.css”>
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/@picocss/pico@1/css/pico.min.css">
</head>
<body class="not-found">
<header class=”header”>
<h1>Oops, this is awkward. That page couldn't be found.</h1>
</header>
<a href="/config">Go Home</a>
<footer class=”footer”>
</footer>
</body>
<style>
body.not-found {
padding: 0 10%
}
</style>
</html>

693
src/khoj/interface/web/assets/icons/chat.svg Normal file
View File

@@ -0,0 +1,693 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!-- Created with Inkscape (http://www.inkscape.org/) -->
<svg
version="1.0"
width="299.99649"
height="225.92412"
viewBox="0 0 85.704 64.542"
id="Layer_1"
xml:space="preserve"
sodipodi:docname="Speech_bubble(1).svg"
inkscape:version="1.2.2 (b0a84865, 2022-12-01)"
xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns="http://www.w3.org/2000/svg"
xmlns:svg="http://www.w3.org/2000/svg"><sodipodi:namedview
id="namedview3800"
pagecolor="#ffffff"
bordercolor="#000000"
borderopacity="0.25"
inkscape:showpageshadow="2"
inkscape:pageopacity="0.0"
inkscape:pagecheckerboard="0"
inkscape:deskcolor="#d1d1d1"
showgrid="false"
inkscape:zoom="1.0445985"
inkscape:cx="150.77563"
inkscape:cy="112.96206"
inkscape:window-width="1309"
inkscape:window-height="456"
inkscape:window-x="0"
inkscape:window-y="0"
inkscape:window-maximized="0"
inkscape:current-layer="Layer_1" /><defs
id="defs8" />
<g
transform="matrix(0.9486962,0,0,0.9486962,2.4834364,1.8361818)"
id="g3">
<path
d="M 45.673,0 C 67.781,0 85.703,12.475 85.703,27.862 C 85.703,43.249 67.781,55.724 45.673,55.724 C 38.742,55.724 32.224,54.497 26.539,52.34 C 15.319,56.564 0,64.542 0,64.542 C 0,64.542 9.989,58.887 14.107,52.021 C 15.159,50.266 15.775,48.426 16.128,46.659 C 9.618,41.704 5.643,35.106 5.643,27.862 C 5.643,12.475 23.565,0 45.673,0 M 45.673,2.22 C 24.824,2.22 7.862,13.723 7.862,27.863 C 7.862,34.129 11.275,40.177 17.472,44.893 L 18.576,45.734 L 18.305,47.094 C 17.86,49.324 17.088,51.366 16.011,53.163 C 15.67,53.73 15.294,54.29 14.891,54.837 C 18.516,53.191 22.312,51.561 25.757,50.264 L 26.542,49.968 L 27.327,50.266 C 32.911,52.385 39.255,53.505 45.673,53.505 C 66.522,53.505 83.484,42.002 83.484,27.862 C 83.484,13.722 66.522,2.22 45.673,2.22 L 45.673,2.22 z "
id="path5" />
</g>
<image
width="75.991768"
height="49.994583"
preserveAspectRatio="none"
xlink:href="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAQoAAACvCAYAAAAFbZAgAAAABHNCSVQICAgIfAhkiAAAIABJREFU
eJzsvXm8bVdVJvqNtfc5t03fko4kNKEJfWgCAtILqCiNipYd8uwoBTueaGHZPX9FCWVTYtlUlfhU
tLQK0aeloMKzKQsBBQSVPpAQQpOEhCS3OefsNeqPOZpvzLXOucnlhhC4Mzl3773WbMYcc4xvjDnm
XHMBx9PxdDwdT0dIckcTcDztmNYBXLR/Lx567llyyUXnyXl3vYucccHZctJdzpT9Z50me047WXad
sA9re9dlub6OxdoCi8UCIgIBIKqAKhQAoNBxxOh/q1HHrZVsbW3p5saWbG5u6sahDT108KAcuOWg
Hrj5AG6+4Wa98bob8KlPfkqvveZa/cTVn9CPXvkx/dDHr9OPHN7ALXcod46nz1o6DhSfA0kEZ5x+
Cp7w4PvI4x/14OHSB91HLrjHXeXUM0+TXXv3YFj4KOnQBky78nZJVAABFAppX6Da0EIh7Ufktiqz
dKkPM+102QBVXSlWW1vYPLSBwzcdGG/5xHW44YqPjp94z4fwkXe9f/zAO96r7/jwR/XvD23gqs+M
S8fTHZmOA8VnN63t24N7X3Y/edZTHyOPfdRDhnvc40I57ZQTdX0YRJpWm+Jq+2i/BFAbKpUGCJjq
Lay467eo/VCN+6n7rWatmk/3ALHfok5D32DmKdX2SYHViPGGm3H4g1fpDW/95/Ejf/534zv+9u3j
6z/xKX09gBt3YtrxdMen40BxO6b1NZzzkEvl2c9+inzZ4y8fLr37BThtz26slUwCQE3dwuCnRor6
rMHAQgfINh4FIFVxR0cNqy88Cy9IQCHaMMqb11abVHSptBghIkmjOO2iSacCagCoBHgAsLmF8aqP
jzf/9dvG9/3+G1dv+Mu/X/3uTbfgrUfm7vH02UzHgeLYJdm/F/d4wiOHb/rqp8vTHvtQufuZp8pe
EfMUSIEsO31qveTJlVwBxWBA4QDSt+6eCEEGeRQ+9WjXuS1S7KirXS5tKJJ+7eug6Y8RV8sK1MoZ
LpK3NOmyHjikW+98/3jNa9+4estr3jC++gNX6f8HYBPH0x2WjgPFZ5B278K5j3uE/Ktv/erhOY95
6HCvk/Zjr4iKNhMKIBW1gIFb1XQhEDdNoXotcqBoNt6VHh6NQAwltSXmqaj/E/XNtOlNMmbpjID4
NITJR5Id3WTk0DZF4WnO3O+YDokUlDq4IVtve/d41a//4erPXvMXq1+7/kZ9U0/W8XT7puNAcRuS
CHbd9Vx51Hd87fCCZz9VHn3eXeS0hegQgBATAHaxzQIXgHAtIlUhVz61rtXZcg2lvHDcAoj6piZa
oA4S2g93p+ncV7stlI3JCm+lkNBgzHFuO69BDAyEyK3QlXW0T+Nfc870xpv08F/83fjuX/q9zd/9
y7eOv7oacS2Op9s1HQeKI6RhwJ5LLpLHf9+3DC96xpOHR5x0AvYJIB47AEARxAwahnkNS9srNhUv
v7jepokTC+7eiscESJs1vA5NkIKmUioFJ6Nt9mwifFpnSQwKc14FOyhAAQFMs1Tw8GlNtKHevZgS
OQRHIMXS5hZW//je1VU//1tb/+M1b1i98sAhXIHj6Zin40AxkxYL7HvgveUpP/Tt8sInPmq4bO9e
2QsgXOhpSg+hCbsHHlNbFAM8oJiSL6TjVEc0ltOQXPmYti9FixcNLCIIqjmd0KqLRWXjd9bJ3kSv
5ZNVkC4fC1ZPdfxWGLDN53VcFQMOGWxaMiu2itWWju+7Uj/xC7+z9ce/+cern7/pAP5xJuPxdBTp
OFBYEsGue10sT/zR7x5e8tQvHh6yd5fuPnIh/4eVzSxgxCiULLCQh08eRvj0PqEfZ9qyXVP99MWD
kOytaAOltMVGAwGGe0HuURQHSWdWO7j60lbN4nixDX5QuWx/DnpLYa9f1NhE0xv+N3aWCVabGN/1
fr3mZb+++RuveePqP2xu4ZPbNXM8HTl9wQPFKSfi7i/4huH7//XXD8857WScIiLpjQMoGjSXhL84
YPgngwVJfVwfvJFSh4qHKDlOoZyF2h7b8qQtnUIGa56CgzBwsCmRFKWHBT0TGHjqEFmZ/H4K1XkW
jp/99CPIj6kZw0m7myFOIXq04KNPyGp8xPyzGIIGxKrA4Q1svfGtq3/5sf+88Qtv+ZfVr+H4Cspt
Tl+QQLFc4MTLHyTP+qnvH77/YfcfLlkssSgiHXPmI4AELJ8ULQJHAHiJsl0ipdYFeQLqkYVJ9erl
kjgy3w0omvI2oICO4X2U8In9iXk2GT/NeEOLZ4jt5qRmjgQUnlfQx0WzKw4cxSvqKtkh9SGMLCFJ
Vz9vMUDyrezXf1oP/dc/OvxnL/+tjR+5/tP6jiM2ejwB+AIDijNOxf1+8NsXL33es4ennrBP97er
Uk0oQDMJmfjGbvNyY5EV4C2RakrPVtOXLUOABwAVKFKjM6kIchY/tmmJjKiRCbe4AlsZoGmFN58r
JdK14zs41cy9KCKPdhEQr2t2WuIsmZmOJPhOpxsMPNKjjE+TaHonAQx+yej3qZ17E1Ensn82rdvc
wuqv37H13pe88vBP/8O7x19vzD2etkuf90AhgvX73B1P+pkfXvzEYx8q92/ew2xOVHddptzRaYm4
0QFF5uGlUr/nUw8HCm57psHY+Tiay011GFDF3o2uClGFqgdSWxkGCnbXHavEgqai5OFoDeMmria/
pLuXqyYEkNTbskuUKu650PBLvETQwgCRHZCY3viQlPpEyQMDRoVeeY3e+BP/5fBv/Obrtn5itToe
y5hLn7dAsVzghMdfLl/3cy9d/PDdL8C5Ijv0tZ8qA7bpB4wds9yarIL4asdciiVTmn4Ur8KJ4An/
SISlijWzL6YNuk2bvjmLrCt5A8LXomqeKtHSZGwzl1D80qL9GDCNTXQOB82CZJtpSmwro27x9Mi/
StwpsY3Snkbe+RWrnDLdeLMe/sXf2/yTn3rVxg8cPIz3z2b+Ak2fd0CxvobTv+wJ8m3/4SWLF55z
Fk6f8QtuVcpnIFA/SY9LQI2nLXPyGK43S/qAfiNVZBbF1BuWGS30vO6HpyJHjCQ8GA51ZJs+fQEQ
wc3iUTA4KKnldkquUwDl6ca2IOHTjMnUyTyHeBjOXQWriUdYsi9RJZURuhjP4HVE3nJIN3/9jzbf
+NJf3viBG2/R40us+DwCil3rOP2rniov+PcvXrzwjNP0lB27FpuUtncVlCWon1SH16+1uFtkNnt9
u5EHQEwJFu5fI70IdHNzmm5khRUk+DoR25RkgOgCHj9BKFuXnXeUFvOdqq/h3ifE8CpJfHdApeay
fM/QnJ543arNQ4EOdQd6nwSQ4Jnm2HgrPTjqMDtEUr4oDhzSrd/58803/eArD//A9Z/+wt42fqcH
irUlTvuyxy2+/ed/ePm9Z52up8pAZmO2d72ESJVsMzPuqqrMVOIKMBd4Y7NZbpA3Easeg931gOUM
UGivUILqaczMmxwEnT4FVAeIP3mqrkTa1SLUB/YqapxFaarCngEDRVjxbdU76c74htpmMeKcCsT5
tE1VbQ/W2OIPaONXllQLe9I78WdhytSFZ2EGUAcP69arX7fxN9//Hw+/4OaD+s/bdOjzOt1pgWIY
sPcxl8nX/+pPLH/0onMWZwMIYYn5OwDW3KZ37LZ6Is8iXPs6aQnAYEfDlY4dk+08Cd6tOQ4WYGQy
xGjOVY1amccyHEhm7KEKgYyV8fYj9kA9DguuVAf3rxVQEN9iOpPlvfMSZTq6rNKdwINz+UE7savC
FTvapL6IA7YmSnmdRbq1a8dpiVnVZCqko0BHmxoKcNOBceM//t6hP/jJV2189+YWPjbpxOdxutMB
hQDLS++Jp//ayxaveMAlcrGo+BNDKK54TC2yYEukLTFtcFBghdFe0loRkRll6DMReLBo+jLkOISV
TAk1oJAR1WPwGAbVU3Zuet0DMsyvtQ5bXWkrIFnMYcIBoV2e7x+DQ4lPqIGoprs/O62YpHqdsdY9
ldmcCgvIDvGbH0qJzWZlvBG/fbjLlAQ03GFIfO9FgukgI0SAj10/Hvg3v3z4v/7aH2/9oOoXxnGA
dyqgOONUPPBlLx5+5uu+fHj0YsACo5oLL1AMzZWUsYsduIcRF1Dd8h4kujT32Ld7JizOvZfC05A+
YKlRSTjdDhjKtDG9XPEEKKSBQUlaaXbscC+I4gHNW5/pv3sQzgajU00hgwMTJ2K6xHlbACM9F5Sp
gFj7ZXpENz2usVOSeMjM95t0cYxwksjHMaJE/A8YV9B//hCu/9aXHfiRN71r9Ys7t3rnT3cKoNi9
C3f5lufIS37qexfP37dH9mRkiw9z8T9PNBeITUo6c0+6a6TTITzuYcxci+YkQcWzlMo4ZTAtniyn
f1vT7ny7C+6HzrC3RP1AalWQqPnQmYazpcAoJRYQz1C4whXa2VonqPm+jaLE/p3df2JB7xlw4idT
fEx7biTAzlccKxtOptZm2oo3TWdQy02olOyLYGbIAWxsYnztX22+7Tv+/cHnffrzeIXkcxooBFg+
8N7yzN/86bVXXHIhzour8SmYV8SWUmjzr8hnSDn746aybt1ZgOGK4VpO9806ler7KYPRm4pF4Bb7
FsiB91iLkR+bt5Sk3ECjPFQZ3Wl5VelJ0LDIWYCNafInezxx00mxJkAh4YPYMyuKnguceNky9jxo
ngvKoDDZOh5eU3edGgs1j3FOr2OmtvqDMjg2z3osCnzyBj30o//54Kt+5Q82vwfAoZlcd+r0OQsU
J+3HvX7o29Ze/sKvX/uS5SKicth+eKs30eTCBM891HDZ/YJmMfutFjGfaABYrQSTaUYY9k6SIvg2
5HxXiSbYiocfFbeN79zAiVYt/IQr33Ql2rMggMJpajxha5oeQmHlJAU8Rt7eD1PrmHAR2B6HsMp1
B2fxAtid0QVEB6iOQV8BsFnU6R+wQ80XYLN9N8NRo45Fy14eVWzq9GjUN//zeM3X/duD3/Wha8bX
bNPMnTJts535jksiWH/0g4dv/vNf3fXqpzxq8YBhsEVv8KPFfaEVYnoRo10VJ6yVuJV0RfN2m/XO
TVQCP0ICgvw+CMRl0pXUM7l/Cr8+kNMg8V94LQDaQ12+ygGkFgtdqc+SuhfjNMQu0ui5e1ppzYkM
K8YIae3VRso9B4mCq5zLQcLm8CKADOTCE9WBwVZmEKax8S3As2vNjwSMx/Yl+RP8m5LfgBo7g4TT
E995qzwE/qRuDLtTK4C0QKecd5ac8I1PX3umAPf+23eu/ieArW2avFOlzymP4sR9uOTfPH/tFS/6
uuWXLBZYpJX0hQ2zTOJ7AFI7tKiTzenZ/TQLp2HdzD8gAffkwStX/AImXiE/aeSRvrJZySxoWc7T
4vJng17WYy5zwUBk27TEm6qUm5XTQgMcMI0ZicUndBRM3PaJuQ61Du/dpxc+RUrC0HkVCcYTJ8B4
7E+qOh1tarUwlqp5AQNaJHZ+45UrdmUSaIxg9O8c7GQ5CI45ujpPeYXFx3UmjaPo3/7j1pVf9dID
3/zx6/WN27d650ifKx7F4oH3lK96/S/s/p0v+6LlgwUyRCAOOdiOFWFNABrYtNSdM5GDLWnXw8bF
7iDzKAgkxMuIC4ltuZZwL8ijcNeZ0Wmge+Sjdv4B3AMIL0Aie9QV1dgXkQDAWBUN3NR4unXyiEsX
o9AJSUxbpXEIvvgVUn+3tKWxdBXE+VvnKrYgVdsTqj1+H8Ej8If4Whf6Pswotea92tX0INJbsX7z
6tmERT4u9ktUzj9rcfK/esr611x343j2O943vm4H8j/n0x3uUayv4cznP2P5o6940drzd63LWqxn
eySevAqnNgwqgJguwC/wOQqsbZli+mH7DRhwcr5Pbi+BR1UkLR9pBS2f2k02xQ4KJfkuTWVZi15E
G13MofUl+eLxC4cI9wPCp/AzNGk1hFvhwCL5V8g9nASIPnWzhucEqdQjgcVk4bVw0z0w0fSs4ilP
DInBWiBqssoyOdTGyrBHNZdS5623xCQp7ZBX0QFfNpjxkM0tjH/0vzb+6et/7OAzDm/iitnGP8fT
HQoUZ56Ch/7M96698muesrxMIJIRcxBQCJvKdl0RaO+XJ25EwQgPbjYQKGLlwKCK3NlJQCBa5CAF
lIQ1/4Evfbpitw1M4lps1famdw48WAKb0ngVvqlIwo1gQFHLY1bRgYLwai7FEyCs2UHHDolA4kgl
ejjxZz6ER5JWPNQBhZRueyVPhAuFFgRfizPBdcx5bl5nxywpgMgPo2XBsA1KcmCX3vmB8bqv/pFb
vu19V935Ap131NRjePC95Ll/+srdv/HoBy3vEZE1DyBxEBFAijEsGy0x8sBu1xof/iJNEN0Q++as
JjvNsochiecxhvwe10y5eeUg3Ho7vQoC8Wh8eAQdtX6dHgCrfSOX1v8GaTwaYPEatIOtkl3wU6zC
s1FJ54wq01gt2NlqVC8nXRjp7s6lDK5O6+Ty/m8JTdIDYbMKTvz0qWJlFmqZScCE+0P1AkkH/S7P
CrqgdDIoJG+tjAACOeuUYe9zHrf2jI9dtzrlnR8cXz9hyOdw+qwDxWLAic954uLfvvYVu1925sly
YjWs3aD5IIZl0BhIj0kIDaKahQ1Fn5HfvGx2jJ0Vdy7g1rh7DJw3dilsl7R01xdJM9yTiA7a31DU
IeIrZelRe8RAW93RrFNoNcQ3ULFDBN+wBWTOSdRihke1T22axjEj6k+ysNyJSYV2+YsCJ5hlPGAo
epy7LQlYrb8MfqHj2rfJlM3Rot3vejk2vbn4xX4ON12SxgQoq0kSvzXGZf9eWT7t8rWHn3+mPP5P
3rT1B6p3jj0XR5KZY5r27ML5L3ru2st/7NvWnz0MZMqYGvN/ef3dUwm8JWKQUpq7KhoCqBw9Y5Dx
Os2F5Fk41W5gshOb3OWVUio7VGuLYFgzkaaE2e/YV6FUk9i0CIriOpiHIjHNyelHgIYLdgliapAQ
rBFQUFHBG8NKb6SpSBkZSbVJrnQNbJPE6RWEp+Z7KFwpVak+AoH4VgicTgk0v2a/FJM+TKYe/Vwt
+m7wIV7ErpJXGEur2nbTxpTE6h9H6F+8dfP9z/yhA087dCc4JOezBhSnnYQHvey7d/3iNz59+XBx
HzEGZroHMsAilFvDyPnwO3BEWCxOpSIgsIGO3Ymx89JBBQit6mxmQEcBijqpLcfFdXWw/DbafcOX
T2FoEgvvW87RWx2+xEgurgdGgyFMv/vp5GFYUHDumP3cC+pt2y0KGBZ+iENZ7lxloJgPiFIHOwJi
67YFcwPgbVk0j8WQ2i/GNP85s+8i87FxwDyNxOKJZsg0Y6Ej8pkv5c4JD4nTmtf0re/e+viXv/jA
V33iU/rXPbc+l9JnZepx4V3k8a/+yT2//ozHLh7QFhR7S2tWvhto6X9FgM6vTjcwNX1qCpmrI+Qt
2HQl3FkBhIIiYeE5yU4SxFfM3bSYgwTtMBAbZsunoDsLaMt1fDIwEGJO1NF8XPG5sWHyoLGaK+Uz
fJjkj9M7lyRHoJpfIWWM2spYFTDrxtH75R6QogOJrrbi7bgIxIaXQm69EIYH4VkEBZ0j2K7nWLjn
VgAFLkv+o/WjPaLejZkCg2N6q0vuctpi/9MvX3vmX71966pP3qDvnPD7cyTd7kBxn4vlK3//5Xv+
y0PuPdwVZcC1CIq7cpns7hyyT+RhxnoEoqfSRoUZyTRrhohPNKHrxcsGuiemWHLUNtx0qOS8VQF/
2jUuyFTohJ7viMemx7wW1nhWl6tShffP4IFcOmVGtOopmiESOxEdYJNyWpaepYG8lKRkmpM9GwcK
IB9rD0CcB64EuOyn1xVeZjTNqyuoYExyVSpnIJsBLAae2LmqoOdVcvrk36vBA04/adj9tMvXv+Sf
rliNV1wz/s1sR+/gdHsChTzkXvINf/DyPf/pbufKGe2KK2nCuAvhdoPCr5DLjU+dclFQE0AZT7dd
ymBh9/xQ2Yw/pejn/NOGtIS7NcCH/5qFmqAY6Y2/3KcCTMc2W63VVBCairBwZjvcN1LfwADaWFWU
jmgVWqQkEA2aWCOkLbzMKheDhFa6aMALzT7Vadd9D4X3Hdn/7XCpJ4PEKWeVOa2J3rt9kY4jHXuC
yL6RGBuSURqvio0VvLidAcDJJ2DtSZetfdGHrlnt+pcPj2/cvqd3TLq9gEIefl/5lte8fM/PnnvG
cHLKFwlc5AwVQ/EMQljp6UMSTDfw7VOjqAOCW8eokF1OgVlLgBvMI+89Yo12LdplCQpzn/aFXG/h
rNnZJMI/B6Tli3oSJGb1kUDLp2wiiLNrciblMOkakyrPtHHT6WwlKGRwGT1Xg0/S0Zq2wNcNiE89
azjQ6hujbAoiPhY7gITX1eNnmTG6onIfOrYLM6MHwAkg7kxEAdvASK3VW36xH/v3YPm4B6494iPX
rva864rxDbeixc9auj2AYnjUA+TbX/uKPa8481Q5oRfGqZi1776VOh4UmsmVDG9q3Q9E/AYX8M86
sSy2TnL64bRo1N8hkyfCNPckTAbnQcI3cg2p3G7uxOKaQlWzl1Mm0uh+O7OGERhGyGAPLvnZobGB
bEo7GDQ6NebP2bHoUoJC1p87KDOWxHuhCJLskxbCyBuJ5clbk8jmTIKNUWEH5N34Rz3oyrM9m6Cl
/ZTkqhA4kCMVw5hvns+/fbuH5aPvv/awj10/7n/nB8c/v5W9vt3TsQYKecSlw//1P/7d3p8+4xQ5
ATTAvj8h+askImwuJtqed5jZQlbGBk9nitYnC6UMaA54sTs5TRGWAQaKCBbYz/SViiCwtAkFSXlb
euTwMK9dG7xYu8aBSQccDAAWCgwNGGQAdMh6WtzHO0p9KELrX3hKJzsNQ8dbMU8gj/bza76nIepU
5quVVkT+blsdDRnxqcPInf5CqSury7Qy7+WyNCu+s6a/NgGNIos0jsZIBgWBxSpYnCjt3S3Ly++7
fOjVnxxP+qcPjX82z/3PbjqmQPHge8k3/Lef2vuKc86QkxKZSZhi7KuGF3AOSUqb48mNupcKEGCk
jmTlI2Yg9RbpR17PvO020+x5dMYNLj0wWXEJ8ofIXBQlvsd98NJg9rEBrVJAkoqItBC672aVzB+T
L/dYnMLCA4HIABmEvA9JKxjdkuJMzW2kytWA9CBmV0Bi+lOVv9RTrlM9LcKZoOPXhMpy97Ru8BXL
n96RC0AvZwSSHSl9m+i7SLLCPcXY9XWufGlPsH/XYvmwe689+N1XbukHPjre4Uunxwwo7nc3+erf
/sm9P3fRucOp7QrPWoEp12264QgbZYB+BFSmV/NTEgwsKdUXOYv5qB4DJGNnseJAy4S9UZLyJ5Ue
9dUBqZkLgSl1Ep+ZyWJvWWsQINRwPpcirgVUX6LL0GofAF9ZiviMeHwEBBIdMEQ3UoF60GFQY+UO
ZXOdhHUsmM8AJ9nG7FIXSK+lXJ71Gvwng3riYfvkOIoZtOqx5B7dOUn2Orqm4271matRi0IMdvUf
nLhXlo+4z9rD//Zdmx+95vo79oXKxwQoLjpHnvIrL9nzsw+6ZHEOUHlRVaBPQ/kuSMFNa06q6EIk
gAdFfeNPro6QhAfTtQ4UwQhPF3I64l4B0D/0xLIZriqQeyd8g5cLFymxhjJrmYN7X9XdcK+XZzhF
whEgkclAwa27V2i7HlWbZ1Me6ZYU5OApWfUaeJbOk5oBCgeIgbDAyRaFDjmFCsDCQMBFD11NAj3k
svPVPg7B7OoAZMc4R98cKgK0YZSQkfCSvd/qYONt+xgKtS2ToUsZoaatEhHglBNk/ZGXLh/7P/9u
8x033qJ32A7O6Tbq25hOP0ke9tJv3v3vHnX/xYWNgbkZ2nfuVVBnhdaiC2OUAQlNFw+IJPAtz5Y9
2ygexjZlXeio+v6s3FZvCuY8JQwdSuvkbSm01y2h/rHy5zq7op0ujnDnvXCRcx2A0XZ4jmJ/MPd8
hCitFqnR59UWzmoqlRNrAOM7NQtviW8xxeB9IfSRBfPIALWTyFTGNvpUeeebEVud3gz7+b3JDIew
tCA7kUVdnL/JFWryqoBGZKGCvG2b61TkCeiexpg52jSp9Wtgubf5030uXJz86pfue9UJe+VBPZWf
rfQZAcWeXbj4Bc9a//FveNrygWkEEwTKhh44n7Twj8cyhICsaQ4K7VZ0BR+9XimyGl8jQFbbn6i8
SqFp6stOFnWnMGcezXQPQivpHk/Tq2YuWjfssXmVeYAQt8j+M5XXrVx7EI3WJkh71F9nwI89q0Lj
YHIXagVGsdMEXeCRW79HweiP0MMtKAq9qibuai84GinAqQIZ0/rOxa5oQMqHEULeg6SSs5721XAS
B00KJAr9zmYm9cZ4KIIn/J2fbs1zj40nIxk/ylOIJRFi8PY0COSye62d+ds/sv93hwGn7NTN2ysd
9dRjGHDSc5+0/NFXfPee5wwSnlUqPu0+S5tMvwSxAcUL54ywZXBmDWZ566qJaxDS3Md45wpEUTjA
VjQ6U2M3qyVKOkEDF01pc5v9rrBsmya3AbcHnfqpkQFiKpvVrhKWRJyxtsQTB/9GS94331ClhW9K
j8jHcx8Q+NOr6qQCMe1xZSqATCOIGFsCKwcVADCwgPE6g582agyE3gXyAGv+0mqYn2JhaOxDJaWj
VZD0RRkKLBtN2pflQQ+yWOm7PNQf7cuHXNt3yy9ETzQiuUIC2PAPkAvOHE4693R5+B+/aev/xWcx
rS2xfrRAsXziZcP3vOql+160ax1L72D5T8yK+OfMf7G92RHV/2KwJBUWWnCB80fWGeEC30eMk6UO
2iP+IQnvAPzMh1pdLnW5xlW0lMhXgcisIwAZhWIKXacSA5qAhhDaNd88lDiIABpdAOMi58lEc3wG
vzRjI6o2ZeHGM78rinBfmbnFQc2OpFfnXkfeF9IwtZPKa5xFynbo/oGwXiAmMtQBkdfrUwOuasYd
LCvi7ilUWfAt46jlmN8l9pJ09Xs5AJZhumY/FwsMF541nHvtjeOpb396RECgAAAgAElEQVT/+Dp8
ltIpJ+K0owKK+14kz/nll+z98fPOlBPbFZdgzMCx0ojZb8rh5w80QEGbo4kayHSSoIMPMVUpGNQe
tumkJnwHQQ0yztFIxVtvWtsJWkYjkK40lzEByvxK7bhDabS7i1tAokMI68HkHRecxIHBBNUfQvIp
jNPqYktSyF69/zn0VctKATriXdM/8nAof59i6zTR1IeI1V+KxDwx0KBQxzTJ9KvTm3bDKCM+RjhB
4SGl2g/GSkUntfndASemnkyA5Fb35hVoyGJ1MGsnini6t2QE7N0ti3tdsLj0L/5+8wPXfVr/aYYj
xzydfQbOu80xipP340Hf+axd33fpxYsz0iA71wXFzAF2HSjRsgknXEL8WgqKu7eDCga3lrqA6pAv
/3KrYwGi9qdYqIZVjXqqfUvajURRQEaxjY4KGX2KIGU+q6T4Ic1C7iJoPg6JOsTf7zEn9MELAUZE
YFNHbcEwfzDM++2AMCp0BWAFyMryjQOwsnpGJC3uxfgrTq1usfrVAMd/l1idA50xyp/STUtugVQe
fgIif6JSI/hKv1fI6yrZVy9O4jPLNsxMC5w4rXkjJOoKPomTUNsBEpLeRMQeMHFKtf8i1ppQsNKf
2hWSQ+6fRtEmN2Ptwj3OW+7/le/f9zO71nD+Nhw5punEPcMptwkohgEnf8Vjl9/1rc9Yv6x2yhQ+
elakq4f3jrk9eMzUwdW5TqoAWJRVCc+bgb4GDAORFiSZ+z8osFAtEegqoTBlUsg4GpAoCZKt7Xg5
sprOGyGwcPfeKfFlwYnEOfCyefLdk2S9oi6rx4U6lDpwuAUwVRVYgYCClNMUlE+JymH2Gllk3eNi
kAfqS5ZrMK900YHXPQxlviGUMjwm6eriVgiQJiKVDeZHBJVcWKTKqV+bC3T6fQ+WG83uDdVzPwiE
/LPUo8U2RsZRoKsOTLNuueyey7N+9l/vedUsM45x2rsu+2/T1OPyS4fn/8pL9nzP/r1YY/nNRH7W
zOUi8zBUdXcMVGyyyKzpkpXdhbCBzD146S/w7kLk9DMjcJV+0kXSyfzjZtVaCqEcwoPguXsYLHjc
pusfQILEhJhXEVOv7H/sI7Ey8aawQfJNOkPzbrLa3hQ7f1I5A6/CmnqglmwJ+cu8WJtQ7YDowVPp
lKJ2sSOny9QVczZ4vyFTtgVlpuDEZ6VMIV7Rn0p+5CnY3eqr71/lPMQbRa1sImR8n15oFF6r9dFk
O09Nz/oXgwxnnbI4+51XbB284prxf0+YdgzTBWcOd7vVHsXZp8pjvvur119w5qmyJ9Qx3Cp/g1J7
61VzP+mP87ht8p2FAIob4APUCdJ0pQI5SC6Unl06xSiywEpK9ZM1krFOM4o1YacBqbhz5MXghvEy
MAtPhdtht0lMT4lwv259iEfkuwbj1SMLBRYa5wBXDbDfStcR0Gq3fTzy06dqGZBNSO6HbMoIcbzr
rqNTthzKok+FVI0xnmy46p2bGLs67n2gl39U4KOy0UkKaM60Has92sZcaZxVBeM4QONPMOqAcRSM
I+dFeLMl/kQNn3PaYs+PfdPeF+9Zl7vh9kx6K/dRDANOfuojF8975uPWLmlTWgkGAKgWci4J4BEj
N4AN921XXv99AESG9geB+JFMXF9fP5AS1juoSldielGyR9BQWQiQys3C4uoCSSe7yY8557TRwQ1b
uu19BFzT+vrlmJ6wD8505b2MN2jELGI65jse7XETX251/S/4bF7SwAYA7qvVbXLBc3GwbBUJ/bWC
U2TozvqpVtwI4p727ZWLztc5r8UAgleLcpOVAZtfq6NCnoJGI9VJ0FmR74EyzWJOUeLULpO5sQcS
B1RnnSR/Y8xpOvKAi9bP+LkX7Pu1KTXHMMmt3Edx2b2Hr//FF+/+npP2ya4cOFqH3qEB/8htup3I
xTUuIzG4R+oAC532goRugDupSwWVSXNcVSHDrbgINM6U04yOS+WKEGF8cpbzw7vPS2gWF21TArec
1C6v2kj2MsC49S19pwTcqe42kKj+Ws+uuU+hZistzKPOhnDF6vc18qlQC0RoaV+mvuXOdLU7+Rh/
H6DlynI6EyMlVF2JPWTBnMr07WqlNRqjqzyF8Ww2hSWMq310vopgsYCcfIKc8ZZ3b33y6mvHf+gp
OBbp/Fsz9di/B/d91uOW33TembK/WAyx8w/6OT/32FE5UF4Kg+s1KsN+7AzzLSQP1bYNWEeFtmUC
QHUSIS8KWKqsVoQFrSzLzeRnqy/+rwDhYxRpNCvpbmTEFQAdxKYH9ng4rcfy2RwZxAXxUC08IfZn
4YnR+LBSYKWQlZswF3+LwKOpRu5sIPtnU8YhxoStoSCXONNg+FvLSkTfQXToGGrMdr41vHb5Gqvy
CS0zkoC43vULbcEiXnGiMe6BP3lNPpt7GsV6aIq40SI2IDytjmk5Y4IVTJ/V6ef8+TcHEi4HsKGU
xYhh1xYuvnDc82PfuvZ/i2A3bqd0JI9ieOJDh+98xYt2f+1y2TY8SPfn7pn4aEw021HXlhOJyXy/
fXVTCuSIdMiqOp0OtAYQHCSTVjdL8T2mMPtQ1vfDE5jBK+m/SEydYnAdcAzDgkXkE7c9I95V7arP
xdbKKVPI6I9kg5O+2We07xog3NPaHRvTWIngzwCHBDLVPuAZJE4DhpDpCjqBTcES8sCaQs2N3WRk
us4bJ1k8Z6exUi5PkIXqShCc+ZsZAJ669Trk/Yxge0fWVO4kt6AvAFkoFmsrOflk7L/uxvGkt71H
j/lGrCN6FOecLo/5lq9Ye+7uXePCA5PgPwAeW2iQ751h89d+x8NSJtD5LKUCGKEY2/KdeQftvQ6p
P/lJpwywxMU0ojbvAhLUMVldfyOSXsyTxHyyxBJc2ewZBPG9Dp7H4wbm4ficsgWsWiALI6Aru65o
1z3gZXQwcMUeDn9wbBzjU+M5kexbxENDT2se617+hfuAzMf7UAI4ORCNMn0QKJ1vwfKCbZMDQWF9
AJ9Ch7Z9PWmVCBgGFYXs7FR/9CjHIsIbCM+A2575jP0yqP1R/uy8CS8fQOCITTyTGQ9kJoXZHKzB
LYFuLLB1cB0nra+vP//pe7529zounC/9maWdPIpdT3nE8IIfft760xaD5gFLPre2gRR/UMjRVNwv
Mis0gWlP/bEizCHblCQWYBSX4u3yW95optGQrrBd5s/OorV/1dpCsbipaQROgYG0OxE2HdHilwBu
MTsBCDuj2b/cjEWfDIZh0UP722+l+sg69iCpOul8KjaXoFWfAMhhLPVCHCQUZQrqPPa+8/co58BS
KKkGfzLM3Kn8Hk6kprXlcEJvxb1dLUTkEqVXENghqHMAdNf7ayEuPm4smzR+ATpSirqscJOleeaP
j81qwIl7h13rS1z4/79963dxDNOOHsUFZ+MJz/uK5bMWS1sULxPkwG0o0rsg2Kxum98jzse+fkj9
jIeKTFijDp7d0coLRYpz80vtVj+WTmn57jLisx+pj8B7ReIVksWKHaSxMSZpbrOpNu8ebIk4tqn7
36DbKlzpQAigorwQyJntVtUVhuhr3ky+ZawCWcebMHoCjPYkqPBQasQcNLwHG6ESx/J8rcwwrCDD
iIF5MGhsAWnXm4INpsNta4gE46X0mcdHYgWxdGROdguvp/zWTpzjh4M48XfWbnlZX0IeiOkhGyxZ
GvfYi+aeTsapH0AF9u8alk956K7H7t0ld++zf0ZJt18eXX/ofYYnPOnhi/NbPjYVznR0zB6BYWUC
4xH9ickgKTQ/XbsBNMKSyBZSG52JSlbX3LUyPq7JfJHrM8GIWZCgKR00pCxhkMZ/RtYaRkpsmY5D
pyKPcpMWv/QApeYZBFDIMGIhrkQtkMjCEGWNR40uVxZJZyN47N2Wujse2cfA4DL/IK/Dr7m19fE0
ntUj82r9yXoGDAZE32ej4alMDbYjI6Ob1nwEHPHcqusdx7K0a4DjH9ZGxJiH5DPo/mRPCedh2QDI
ABBbIgCMyl8an9l42KSvyDIdmfc8b3Hi9z57149vV8XRplmgOP8sfPE3ffnyKxf+AKKgLl2Bem+M
c5rFVkNcMxpoSACI97ZGeanxjOzBNVLQrPHEEpN1mCiJpqKErAT9dYyzW1p+RgrAd2aYlS1PO7ZS
JY7RVxbWKAGvNwy5f0FNvywGwqBTrFoFJ5ZfQV6Lsl3HpyDInpoDiRbaw5tiBkFjGzvvLuSoj7pM
SLYdYBExsLkRkGyGL0nR3XZZ3IITKBFTsu1kXNk4OIx2UHF6eu4Flr8h5db3QWT92Vbwm/nFshqy
mwNV8I053NnVsonHUP+EPVj70keuPXHXGs7FMUxzQCEPuKc8+kmXL+46OtFOmwWU0rumQ2iELBT8
dCc65cnKIuqsyp12S4PBDAbVfW+fsbSEHOhwDGKQtBO5nFLwxihxk+xWsodq7ydMaQkcJgxk6aXb
uVFmgK6G/B6KVQhFrMYUy5WbbXS1wLi1iLp8LhyKw5YKTSldpkIIWfAcpliRw0SCeBPDFhvVUpkz
Rwh3/JAWiA3fHuA3p42RX3MzVKGPvCdvoWc/9ZlsRLHsIXZC9+hBrYG8i1zdQwEW4lY1dKAyQAFd
LrONpQKjQ8hk8T7sb7JTM+Xu7uctT/6OZ6z/BI5hmgDFSfvx4Cc/cvm05TI2++fUDEnXCMFoyr+y
JcixGVuM0qL22q2SNOBo5TmK3copIWk9N0ziH/5NQhsKL1C3AmQNasEuWac0fkjxUlmpHMR8Qbu9
Q2MsecrYzXgWoTRsJXVGbjxLcQ+831Kt+yjwE6x46zmv6SuBl9OQs0gpHk4TflOKwQBY0ihwOV9J
YS/HY03tlqTFDb225xt4lWcc2lZmxAOtNt0Ue5gt++WKk8u3xDJmOYMGXQvjps7PaqDYy6g2nYyh
oAK8UAMkVFED5eOlci7mICklc5b1rTBO8aRz9vXk/Vj7yi9af4oI9uEYpQlQ3P8e8uivefLivnOZ
C8skvR9Xdj+20a9Ve89laWlUtDB/tHJNFpibbKUJckmRNNzY5G5pdw4zymghB8aFphcQ2MlVPQ3i
3lPbu8/bbHmLLgS2yjDW9rezjMFLyhPP1gAwgM0TsFNpRwwxZSyrFfTJeiY5KjbDsqVJWzYV4mkJ
mJLgqhEs9mh7i2VqDB0HoDOATcHBCGybXEEwkkXlY+gaC93rpDGiMQTIM+lS23qTcBBTwrQMxKTW
/limZiTbrswQ+FGA/KwQK7xPa8JDoTERA9fZlEJIyeVTCYiBC89enPrky5YvnK/otiapQLFc4KwH
3HN49Kkny66CxgxwpLjsYoG/W6SbYzUKO+JSmseR3ZPcfuBdF2CUtlQ5ioa34YILF15/TWV4r2Ui
En8jj0Tp0oxFim+pMH10P/oWg91+i9EWb4dCwZEUK7fUnjebrELYxSGqVnNbyqQZ30mMA2hJqCh/
z4Ay7WNFnJFf82/SwnudohTUNcWBf9YYSHxnwAgmu/Ikj+pn5pscadKNrboSRpyogl60bedcqgns
hDbzcogJwYvsi9NjghBTmzHHiwY+4jM0KDFJnrFqZSgE0IUA6wosm26cfbqsf8vT1589HbGjSNp5
FBeeI5c/58nLRyQlpFm91TUtiHkdKwBAQSoPKlUl9H9z3NllprZtxDneMGvZJ/XNg69bBIlM0lYu
rO2c/ogpfgOi9lCVBjjEto4BobCNWs19CQ5qQnlZERlFPIUVJOEdU0gdRBr7WfGzx+5ReD3pIrjV
T/Sec9+djAw0Z11pSWEynEHMaNc+07qStSAFLQMzGUTub14XRT3uoivuv8nBKpgSN2eNdtde91kA
QNqPmI5S2/Vgopz+FaMQ96yvkztoozyPzmHAgsTFiGHXCFlrfVssMNzzvMXFp+w/Bid3SwcU97gA
D3zYpXJ2CrIfoN9ZLUFc46hyv5LhgaESBAo07RSEmQC3vsky+L/Gu+LNIGMf7ReBjNQh8O+qaVlZ
0kLWB0PppQALgQ6Sm+0JGBSCOFV7GOxPMA5+HYVPk2W3bkA8z1D2G1QqHSzDmqP2MTxeAgv+Pqsj
XXIrGgoake2mwL1Fm5YfUNTCbU73IDCDddFfWw8XA6AA9rn2SOb8vnaDnp4wZm5Ehw0kplOisgXc
S3ubDAJpyYqta/cSOOueIdDrFsSwLKWzADrrHNe9KRhvWUAPS4T4zj1D9n7tE5ff2ff6aFIAxfoa
Ln7AJYtHrq95EFNmgI5hnS5NBqAzGYICGCE0M0pUUDdceF85Iavtds1o9O+EsS7ds54Fd04htFLB
GaNykylDp2HINwECMZ/O8r7zTiJo6x5YxDOCmzKlz/IqeTCTJWE03hUjbnXxNKUI++hBRAnh44h8
8IZcbI+ue3BSkvDkXeU60u/r5MSZxWwmvsSUiIai4GqgvP8k7lnHJ9MqGpfQWzc0buAmDViFnVwX
TyOKaG2nq4aNQ0x1Rnq83LxFXmHM1hSJUPZJnmEnnkU9BYKT9y+WT3vE+qNxDNLSv9z1LvKAL33M
cL+gwCeGvfIQjZMDV3QHKwe3JCzkkgjs1pRGgyca7gIziEQbJuzpBnZ0zSVHaatrRPf4smp4Hb2A
sGqpVMWsPQbJm0ZJxykJrQ79MXyWUk3g2k7daQMSQssOWzzvIvlUYu3FDFj1gAFFOzFdy5bvxh+3
yp4/qk0GuKxEvyX0oEz7VBvA9ha8AIZacaNHrJj6eKTRcDnluoLtQp90XYMYZCW9UFMfQl1U6zBx
Hqszpi1jYUkbG4+dBOuIBpKVAIjOYIvV5+UHQM47Y3GX00+Sh117o74ZR5sGOufsgrNxj/tfMpwG
cYXpnqbrUtiQIC6jxSH02tDTFSkHX6Ojzc3WyYYqCML1DhAh7yBfbDMj5EcAifBHwjpKgE14D4b4
Kzu3UMq97OMgABYCDOQ5eCvaYvbeZks9IBFhgrYngVEjaO2CnKTDrNPNg6V8tG17BNVHw9fGJisL
+ZM2VioWTI6bKRiTGIgFLtupTfk5N0Y5hayuNJAyEOX8ugMuW1UGFaVPysdkZn3eBwQ/HCQiZtZb
O/9k2phGJ6sMMhkHUmYvo30/g0aNvzj6JD4bYA8Lhb3AvsqvVXHWqbL7Sy9ffl1P6m1KovGA4a6z
T5eL9+3GGmBr7gD8LEiCsPJdjCIXv+goSPCc4RSMC0tHf3El4hm+/8K/a8yGAqQmHUJOIWZuRgkC
iGzfRm/0zT6mZHZag7pQgyLWNrWAKjCOkNXYnugkClO0t6GbgTYsilTrwoyiyuq+AtTv1FmV7Dt/
VkedGNMFWttuxdGqFrRNUwNVltKuWUXdDqMZdapolGQ7aPH4uReapTsN5Tmf2y0XxQg2avvOgDRh
hoRxSJzMKW4zaIixT302Tkru2pRuushT6NqvjhCO5S00V8YWOQYyjJCFfbpg0I7W6KEZjZP2y/IJ
D7EFis8gLQFg/z5cdNl9h3vnZRZrfjpS6jhGQK/2tyad/FRyx4SnOFz5LA4YI53G0CKiyySFp0Be
XYRHRWLFQJH4F260b3EYxgATjIL0hJCSEo1mS3mMXXZq0h23SnViOrnf6CV+ELD0EbuYBlEVDp4x
1VPCG3N3IWN2BYhKBNSET++MYe27xENjsTTJHWV6nL8qc9wgKytk4lFZO1/SZGqSGTQpSnEh+VJi
0ly9uSszO9dT35qtxPYj2kScBnTipXQDN+P1hBH0aV+AWy0WS7JoILO2huHic4cLRbBPFbfMdPNW
pQEAzjpVLrz8AcsLUvJzf1jturv/fmYEdzYFMpC0noGfNZF1jF17vsmCgzU7plr/YO25MEjstcgV
Ee/d4KbHBEHRFHYcF+YmS06JloqFb5AZABmGnGpQ39OPXkBlgGJB9ojYxBgCvrHNb5LBDD46j5KR
Dmb1RCZQIC0tboyD84RNJP3xNJADMXG6l7IJIXI6ulv7yOkigVxWy8DPBZHWfhY8OiCdYSEDq8ed
fHx7y148Azp3g/NlMBQ9mm5jLCX6rUivq/KHppbdYbyZV+3wZLUqNdXF9WxQyGIEFtoOkVgoTjtF
9t7zguGpc5Td2rQEgDNPxfl3vwCn175pATk3V2UNQeYZwygbVi4MhUyKRhXOFBuBnM5MG2GPs8iJ
0BZlv26N+SCx0xLQKGiC4ddE0UYFgD/I1G7AKwvZ6aXS2hQoPHrVY0O5QFa7zMtKldJdz3oZWPmM
LfHImZPkil7MrEQQzqdXMSWz4n7OQ1Fq+D0FZNV45IJNXZl0g8vzPefHbF+TTYEpnREuQuDfqRoP
cFb+ES8oHsHThchbA1BZvxHU78Kc7O/p4mk0elUgC1Osr5ryF2Sah1im2opcrh0QQHf6Kbr+xQ8a
Hv+eD4//HUeZlgBw0gly5on7pJ23VyK35GH0nfEOGNEuoq5B7qSZ2ORAIXhJ1sUBJCUrpgEmuP7b
2xPkqUfMqZSHOqgJJP14uAfkP51iQHREjpDft2tzQlp45L0fK0x0OlIs/jbAC++zBTG3UzYmI97i
RbEhNr0RsKepXPV+G/11FUCzHLEiRnQwGn2oZZjvj3pY2ioLJauGQZwvWWyu2zMWx/NF6DPu50jw
eND6Wv4zHVIqlrJs8uv9JsUV05HwEECGKvqoaYSc107MgIJRKraVx7fUE1iNoUOIszza2Cywf48u
HnyvYfaxjFublgDk9JPllGGQhQt1nYkxdLqJyZ/pMeRTopF8fGTIZatYwutcd9Jm3n0Y617eroGB
S6yAVhscJNRljhSatKjf9OT9nTx45B6BIO6nYli7I7qYxAQJaJmVQKgSkLGaoSpHkJIZnUlePbGP
BmZw3jqIe6YB7ehBwFc0sq88dpKXvB5JvgZ7bTlTCsgbEMgIEXKfYyg1r/kYMcoxhgDohgvwa5rZ
maYSNOzLwVZ/nJVcNw0hs6I+HRuanvLK/IBM2vd+8NRJxN6zuzCmzNkJBjYnIeJkBma+J4PqTr1q
A7a2wHDx2cN5Uy7e+rTctY5T73nhcL7QhFMmI5cKmNrWWNIse8vhqyDV6nWzdAFm3ZO4wkPbKbQB
BAuvYIy2VFjYZVJ9RXuleu2lLF18xJWpNacxUBH8DH5o9HTGBlULxpX3ghpGLjVAGXilnU+hQAkI
Z1mNqtWEScYU1cZ6Td2MaHkgNGJHoPfbfN1grQM+WT01/viZmq64BQj6vm93z/ui+XWi7bPs0+xK
GQKl8qTCRF7B+Bl6UvE6+ZTJlS6m0+4Pg52PCoqRBA0EcwvBIBpT9JAm0QIELcXD3RNiw8shFD1x
7/Lk5QKnb61w7bSHR07LE/fh7HtfJOfJlJIgxB3FZKTGZ4JE5ua63CLQFLDAjzOiIbxk3eLN5UCw
YVUDHJ4T+xQlWubDoeHLq07IUO75SPRTFtjguvnJfjZrCfImgk+a+aog9wHiOswF5wKPc1kYNPUY
nPeMh17G3Vpt32uwL4UUBJwww9ZoTx5U66i1QRf8ADXNNlUQRxI6/QwAinYgMHG0xTmcGLKwM8qb
stCtQnCMR+LDeEijy4BCnPGNXOloEWJ5nQ4QLqM8XRfXFwlA1HjqkaYZ7hLZ4Ki0A5YdqMNYePaC
T5ZDpR1GnSJIgIZEdgB7dunaOacPl1358fFPp9w8chr27ZFTzz0Tp7hQV/vPFmImTWSdBq3ttgq3
PMBD6/gECIT88Xo3obPk9f7hGlYEf74in9bLGES+ccxHwOofBLIQyEIxLFZlzbpFkrOhmNZoExAG
gtbfFdJkZj+Yr0CFkNIPZ0qPjoa4Pqdlb7isTnheogJ8WdBiCcNI5xtrtKlSDxP3SDqffcmBeAC0
3yMQI78j134ExlenJpSGKnNaFPMd4GuGIlE/P+EbfZOUCV8VsI1KuSW+srriR6JxYJgX0eQp4XBd
sQhC8qOdkDXGfgvDkJAvz1xWsCao1vg5DMjngUhdYjCt8v27sLz4HN95fdvTcu9unHjaSbI/EDBW
CCRQmGMCzgyO2HYLgO2b5Hjn3Eu8F2GBwqI4uM5Y2tKGeDUUqQeqi+71W3tZY29GxKyg55Oo22kS
RbiC2cN8elO7tkNXnWfEA+ZLodY74paU+hB9F7KHDpRu4QxheMrSdpOmmQlwgfHdFcjq8X4GAXat
DIejh7YASJ5H4Za1zxydKysIKiP6A5AniQxHWd8OOqb5SUtqfi9mKzeNzDBLodzO08L//qu3bVNg
b47HlIa+kOnjU+S8GAdCeC/v9Ye76IPlioAW3OS9J9x9o3f3LllceJZcgKNJAiz37MIJJ+7DHsAZ
mNxVey4fgha0I2XNiDgjMkXQkR5KplRX55PvYzJJg8/zYxiNqQV4oLZnQqKiEtmmeU4NDFI+0xxh
4c6HIiyLmns8TdEH+5EvGiLJKcth6arO4Gr3w2qSfrIyTmIWTmfTpw6wYlrmQKj2faApjnt7ucrh
ZdViHAUIAiwWFDROqoXGKj0gEu4+thLlnJHG99LBLmPPKtbwUDwCTZ+S8vMlJqNxBIKqnT8xzOlc
TmlyNoVEMzKQ/asPPBju1zpPruoHZhruwEIBVSZijPoF7slkPQ5Ku9cxnH+mnI2jSQIsd+/CCXt2
taVRtpltvkYIlxkKEQyIPrtNFc6B5v0LuSPQ5GcghEQ7M8IRlEVmLDxNYnys4oGwXpdCwGj+F6sp
IIvfCWbwoCK9EANC5yOLiRAZQkcVpTqcD+iueR3uTvv9iK8Er8EFQtgzbuMMyXmxBjPYeKVCDaVf
Si6s0esu9RgRzFw5cGGNVzcglcFd43itkQn6qMR71OmiVav+Ty9s/Eky0i8YFWSOFSy/l4KtI+fl
pWIqU8aIzZnVEaKiWVesPuXveI7FAap4TKZBBGbeBodLUoaIpgLklF+B9TXIWafJqTjKtNy1hj3L
BZbFEjKBPECeTPDcE5gBwejc5KLdSLBQmpqExHPWtBRK37lqAoJQVfo9S78PquSLl5iKailMRQct
Sl2Wu2Z0XjxP37635lOGsv7nUJujINnBqIyXW3sBbl+87gRChy0i1hgAACAASURBVB4mJ1YqSMPY
WpZDfky63TOJuuPU9Qa+sYRMJPs0tt0xreynH6Lw5xZCEcUAyoCrvs9k0v1YymTFcvLL7NT4gZ71
wdvu8nYyFx8UNHYjaF891sOY5XnCPipX3NFo36OdKK+ALPKqCtwdCp2SxrPFmsoJJ+heHGVaLhdY
GwTtXCqtrm5rxwer4HZlWukYSDpBVhhlasVgoZSPGVLq9h9S267tkLb6F5nJR/X11UHRvDn3NlxR
onP5VUAobimcI8fdrs1UWhfAqvwucCVuMsOPicvq/aG7TJrC+mXKEWQFNhsN3k2yTkpvg/PCAn9g
T3NlRdC2zvdWjqY9saqBlfGaVkY8eEzjkuxrW/KZZ8k5++3D3UdbiYVF6fm6NGXzoGKpT3JckiCZ
accKGJ902owVJwnvM3Hqgc0HlWkggIpAt/PazmaVQbHAKPv2HSVQiGK5XMqaiAxuYTnAlVbZEVmz
Z72CdYkF2YUjpg5KbRSpgveYBDOVFtLdYi2o+tQBiAmr0uAIjQMpcwyiVj4wIDiYV0itgInJd6l5
usr8oa3sF9culdbIqdRNVkKZuODuUYS1of7lkEu4vBjbS5eCVG9IpIKDn2cKUIDfYyFoG7KiEZdo
bTGdUCrjMWjIvGlyy1NPGs/SH0kCfdoV8SCuTywI7Y+1eoUj7Elho8ODg9EXAphubGPKIianfmyi
ME3eHAEEj+ecHjBSRj7L3OmeAPQuncYXHRSyHKGLRoOMiuXuPH/mtqbl+hrWhsHYT/OpWPEIkezv
ZScmOwkJTwoSEgbM7VcoHDDGiwnbTAyPMs6glvNU3PbkWOTYuHCktS5DacDp9ktpl6FfK+PYje2U
Kb36+9WhBU2F71u/pPoOSWn75Loyg6/kuMwJfG9HvPLQTiYWFsroxxB1BUAKEButJB9GcmATwEIQ
Yod0KNzKxeC50FgdTbFGoBu+WB0ormbynDlXf6F5NLFs7cFZi/rEdY/bGGrK0GgZxWIGWg0J84EN
yNCWJ2kBIvpVNvdB7JUDvWlhpmcn3KNUAexVNQSSWSZ0cFBg0R4/D69mCYxLezhsBMZNwbA80lLT
9mkpggUEMnikWZzBmiip7WSKGAwP4kUAiFKvs6VnOb/qwTl+SV8l75voEFnKr+4GDTaNj4QIDaW/
rlTtpimW5P4Pp3hiVDQBlcmYsiUFjA11fiECAhwlKTbAK/EMBkOFq0O0pSNPm6QBQHkHCFXEBMdS
XPKrT6kHtlQKp4GCoPxe6T7xlCQHBwTLtKTLfOeRQLRdZCOMHBJog95WadalTRsXgAwSr4fkzacO
KuqAqzDPoYHg4FMTP2JwJRhlAO95SWpnpIjA2tnFxtjLDoHZCULhQa0pdF1tNVDNo5BG/JbGifVH
m5arFUaoBNYG5dwztKWy4q6CGiYBTxOnZR1eLWIotuyW3kXHRVD5/IEQRr6mXaGoRytN3DMyg7mD
tLdafJn2Wai092T6EhRQYhAxwL2Z88oi0NYypMdWlXFUCQtYijvHZgY8e1K5JnAPoSmBr1j0sZVW
wMd9CPXlGQOExryhWchHDDtoSmFn2svYFCoDkakUPP4JhN5YjoqzdK7vhdVulSE2/SDvgQ7baaBp
42Uvp5psxVGqUkBLqcRcD8BiaM/A0Wv+PP6Wy71ImQsAUVsJyb7y0729iBe6vP1hbHUtFNgt0MWi
AdsIYGMENoBxc4GtTV1NuXfr0nIcmzyWA0UmituZwV7GCPVLeRrZUEMHAKltxvzcJ/9kKauy8w2Q
lEjeczI9INkP7kSRyaL1ykMw3HCDLbzXq2R6XDAkPkPgS0TblYVBqp9mTMGkBRYrkYVFZJlaM/T4
t087fIpegJlZ4t4M1e71KqC6gC+rZrt5dkP4GMRfiXEDXfchUa7I+CXxvbXj45vlHD5i9cLmQ3zC
exqzlC91T9Hy+/s7Rvd0hQApgCXrib4ZyET/MRBPjDYl0BINcMind5n/9G4PXzlx+SKmiXr8yHYS
L8dmyEWhywV0bbCHFVfw/SG6sdDNg+MWjiKNo2A5sivhg6hpSQLf1RUvVymSITkgTaDzDVVRLzUg
OoYitUEQtFcDZKQ5nRebN8Yux2Sg2vjE8pNjjEZLCWxkqKQIMaW5ayQwPg2T7fJCDUikdIKVmB8+
LCWd58RXVxbphqhrMUhPh9pvmiKopDW1455l4lHQ3hZXpuhCruu3FklofNxlgNpJWRpgJSE7BVQg
0bu46vUHiGsqpDaF5J2jHh/S8NIA6BA61VZIEkIj0KsSfI2B1CGex0grT9M8ekcswktwObIfI3ID
Gnmc2cNGmKTrMLV/ZgTCljloBHqgDSDtVVEIsDLgWKJ5byPsSEaBrg1o71rbwMHVeABHkQYBllsr
3VLoKIJFIC6ZXQY453hcQxqJvCYdByo3RLsAKTOMvI0UAlL49qNUH4JIyj942VxmycHRjLhrgB8l
QSzjFTfbhDceB3e6uXgAg0RgL5VBZwAstILqagLVvCAtVsfBIDwNt5IxVqx0IIvpipDB2M5I5UB6
XR2WhuqbR+UAEk1K47y72sbs3FMRrOzALJEhfzk4cPs+n1E0xRWZ8J9jLrLqPAtvVVEdMsGkowrf
sesWfmhBSwVUR0QQ2IDLHmsKIGn4Qft5DFB0zGlZehfZ5+SMyYQkWIS3gaHxdTDAWCQdGAdgcwTG
FTASoAuwUuinb8FNOJo0KJYbmzi4tYXV+hJrXmtBegJ5ThygSQvjLqYPTg2u8Zjwp5DwND77zkSz
Tmz9+oEttbkg2A7CyN7vRshvMf0RAL5kB6l7gcITyFBb7YTkHyTq7PmWvPAovHOJKhPzqjSB2t0N
t2DJr1SEGKVwxYwXI0WxrF4hwAgaQ1IFY6EyFav0RZNyBgspvztNSEYgkCCGTqL93oDExkwGrzH3
a4grrtQqm6dKkKRA+DKETyJOK42rosUxQs5JeY1xvIU9PJn43eprWNVaVQOQgWISLEKh1dYOj49i
MLAwwBhGyELbS6rUDbACmyvgkEC3mlcvyzYl3NiEXnPteD2OJgmwPLyBgwcPY2vvLh/PIrbejQCM
7qp99mVSMMdutbvEF1AteiohQUxn2aYKTxd91PtLQGOmg3WAUFeTent538Gh9WLMOotF8TiAKzgL
AHcgpUPjvXjdhIHzOjA56Alaw/6AjLm9CUAOlHZWQXgRRq9H5T3O4i5y0fe80DkNcS0UmZUNBmsC
roymMFLq4cihejudUfLrrj8MWE6/kHvAYSu7bfEHWiMpe565PzNo6J4AEI3Hhixhj85BCLaJ0Coy
jza2WggBTUyZEUfh+8lVwYOBjY2mHbBNVLGKvWIaBbIw9q4E2GzP5By8Caurrh4/iqNJCiwPbeDA
p2/B5qknkLJ3AuSDVASpBOZIWni64J/a5RmbQEkwOwdJyaJ03nBJCTCSN8VLE2BEWY36c3MT94H7
UoVUoqIuL313UPF/4etIDIzkGeX0S4I3Uzd9hGi+E0PLEflp/cLdD6s65gYipAfR60C2mfUU7w0d
6yl6335jmhzRxppH7B+l73BQdm2VaZW8rBzXdHq/NyhNzgx8dI5QK9/LldWqtNzDtCfYk7elCeZa
6ql15gfVWxls/7PBy8OFHCRCpVxtbEVODRiwpsBSgcXYeLAacGgL44eu0Q9ty4idkgLLm27Gjdd8
fDxw0VnDKWHdNJ9/aO4SdzWtWQx4Z/O1v0QWLAbC8uacG+FBsA0YrRwR4HY2eNt3imMaMXBddoKj
aL/d0PAOSqAw4g9VIltZIQXT4IlQnompdJ6wkvddifnPnKC7wKYrXYCV/wrPJPgxWSLlpTtS3HAc
aMx7hYhqaNWq3bcDWbwiaGCn2b8CyFWSqH5lpe9yzbFPoznCVJlmLMvw+dhaPq/SC46W32FuPNCt
MjeULWexPi4TSXwsHwvI+21HF1Y983xORPOsZGV9HaXFLjDEmB04MG5+8Cp9+yxptyItb7hZr3v/
lfqpR91PztXoTLoyZIydRLgEeX+ZcZyXlTWXeSy7+jWDCEPpyOEIXebcFE0WtZUSTAaNjCsR4IrE
Ax2iGmlyxoK1HQaSLXMoI4u0F0qBjngEaVdCZBW8TBxXYbXM+EGMj+dXgCP4YgG2WKJlj6YDEsBB
j65p+VoAJCgjl6SASU97IknWoWkDom50bXLP3YcnNyBWi0ByFvxBWN6YRjoAuoyMeY33xgRrDMhF
0OICTox/CnkG/kcvG9Z+YCXHsDAgEFlNVhM0AuYFiEmmKuxILNtRakT7uUkrZPkBuOWAbnz8Wn0r
jioJljfdgmve+2G9RlUu5aBj60dVvE4CrA5bRVDMrCCkpGXwLtquIGSj1Xhtgu1oFOFhXo8BiqdB
U5EYeB8QofYIidNtbWVDqeL7DoklyVGoZ08f7zGh7lrvS02+51kXY9QZAVhFPFjFy3MMYg7+zbqS
VS0dFPTDx4LLit4m3cHkuN5VZ8XJ4iJlV63u3qgER+pAOyPAHkIvp1xNrO6QAYtYjZWJLpaCE7Zg
HKS9A6QzAIVPTscIYBzCK2UZHT1GEWwT+z/BwY1dAGDblwCFeWYLNMTUVk4dJKx9ARqADAnS40r0
+k+N14+KgziapMDy8Cau/uDV4/sUeBKvUASfrLP+nEzIhtA6s9rbypULIYTU1LCz3AzMscYBf0Yh
ynkwDmgeRPBkBjRCefPTXch4iMkb93sk6OISPNFdspqSOlKUi9ZT1UbaV1BqINUBU6J95fowk4If
U+9HNAGCn0eIZzrcsjkPg8y0uL3yOFcD7NyiMn9HQU8sl0Y/PjR99YCerwTEfhr0Y+J0S1RUd8QS
vyXzOxVejheGeGqQ4kC8cEIYC6TVpsMQr/mDP+laKiK+e1F63Z9/8cMAewPZZKMNlAj1yw2dajMK
Cvi7EkTZSBlX/f00osCg2NzQ8X0f1ivwGaQlgI1PfkqvvOUgNvfvwVqNwUsyqhcmn2h6cMUtnShU
/ckQXjjNXYoQOySlTDVamW5RpEFICLYzgkEGFs33Jb8qCDqMdX4cmC0FFNxp8U1lcbq3EsSxEIQA
ulRR8FCcPdZwmMgURpLnVAbAkbnkdbDMq4Af7Dv6SobxAAQW5c1h1JekgUK6rBiMAN4PJ4tu5b4Q
Hmm/SSs1SBD1pWZe9fDbAWLFE6288qtKwdImAx3dTpfRnTZNqU7ql3ZvOGA+aLPmWAHDoMByhNh+
Bgz50qjaLTI4JGoR/IxcEoXF3Ag1I+NyX5aHR4FiiJ2eliP7IoDKYBvG2vtJb/60bv39u/QfcbRp
tBcAXXMdrnz/VeOnH3jP4bS8WweSLaafmdiSPxegCRj26YE+dYa4ymngvUsGLbUlF91oSCHEytNc
ODe6CPw9mM2C09Z2c6GFR8laaRYwPxGDqVG0T6T7iJGO/rgHQTyLLhCMKQL4mEaeHvij3t5bt0ah
+OZViAODg0MAZ9dd543RmqBRLWHZUwDqhtNjXU4cYtPivPb8LEti1pJA2jIr5QF5GR35cNY2dlqL
hMDuZc09KlkMTGByMR+lTcaM6kl0jHUZpPgEN+jt8WeYF88eW3Md4LWM11xcKqnnTjb+qbR9INdd
O2785T8cev0MO251WgLARz+p7/lfb9cPPfDuclpabyCWwwRor7PPjSO5P4XmwCHo2cDIiuKBnILY
jemJ5EMKYPUZ0bMohEaiBgxYJXHm7YSbK+g0vJN+rj0Aum1mCTdXEB5MKat+UzvitFwXDMWDKOIZ
wVrfQ6KhzOqNTwLCVr9qrpKY0Ma0hHvLw+FZ+7hFILRGdSEWoaDYIWXGQrILfAEPAv+StwPYWkXx
MsopdrO0dSBIZTvDX7wCX4WLA/xE483icEs/LqJN0Y6XVql7W73URQ+1jo1Q5s6GIBYcuDC67oU+
tp+fvG686cMf0zfOcOBWpwEAbrwZ//S/37n1ZtWhWaTYt26aRUE+gWIQf02eCzpbEI++ZxkgLVYN
GNq98AiGQOOm5zkH01HS9TcFkNHy2F+xqM5gcUF3xTNGqkJ0bFtdVwBW2v62FBhHyDhmfiRABqLF
wzBD0m4cCb8x8hur+a3fdlPUGcFWV6vQ8TTC+ikj8cQeHWdDN69mxHPqSrNS9KdEG5LfO4NDTaF0
BFThjTh9ipxjq5TvPNZQ2PMLKKs1fv7M4DwKD6rjQBkH1Hp7RrHsAGgb2shij4BuLbDaWGB1eInV
xgK6NcTmKo9hyGCgEp7srUjWD6e+wwLrm6ZHg4x9CRVIIFJsbGL1nitX70VbBznqtLT2D1/1cX33
DTfpoVNOxO5qXayj5iqW94161JrQ38GiCZrQeCRsFA8kOOBWyHvsEN1+R5sq0LB0YgJkUw5iUPN6
aObsQhoS35Q7LVHCefDc+psnG5ubL9mrwie/1FsV5wDz1f58ai22osHzfoAtlNDZnkgrRHTFCzmo
Dba0OT7WBnl7XLU4SAgNAYozk5ZXkR7HbJr4ABOdjRw9EPXTto6GufrcdLlZ6/lRmEv3AzSdDm5a
GniqAroa2os3jbaBCCpx8ImbMiF0viNC4uTTtN5F8cwBDJp3aHurquCGm8fNP33z1huP0PIR6Ypp
3Aeu1nf81dtXHy6vXHfr6JQDEUmNSHi4+GN+Hwjp0J4+G6BeLOoJ7JTsbJF49kWFuB2WySqkd6x4
XhFDdaFG3VKNA7A1NEuwNbTlLCK/WdABuvJ7HgfIAWiCldDUPIuhPYBl/WrxEv9DWMbw2GwQoCie
kJJVDda6RV1J24E3CrAa7C8tsAcehf9owB1YylF/bMknY0/emuYfR/jJwOVvNHh3I5FCJwWgmDTW
rd6qbvd9x3vaffYgUQiQHNfCq1wWdU/ZjcUwjMiXVbU8wSP3zINvky7vkDLAHEusI/F8rByIh8eI
bL/+sevHg697y+q3bkvrcynO0PvYdfrWP/yrrb/5sst3XcLeQlnkUrcivlzWKzKC2nZrQLycMjE+
wYCdCvjWaqSpKgI2hDAJtxuBBwcHq3/ww0iaprTYAlIhglSJaHP7HFqgzeMKhuwTySxfOnVQt7g1
KFX+uHSAMXXZrulIS7DuzWlmj1UeFyigsK3UV7rAYDfZFlQ0txWVyZC0iuoFDpS6oeBQYfyrKMLN
ir4NGRMW9Xm3vdff4PHv+9ODGImi2qloIw2Xb5/m/ky8HgF87zHLzgREgRjbZH69J0XutctiOmb0
rVYY3/+R8YqbD+oH8JkkIaAYFbe898rVP3zqJjz3pP3D3rLvAEiPQkHrtNwJKd/F5uSll94XFYQz
o0AcDJjiG/eVyrflxoUN3Bj15SO92nk7SHcwBtu/0159wkKPKrd8+f4K1SGD0k6Z9SUtZJX8jGlk
+271i+Xi71afkESW4B/JWlx3GkkJtNQ9tWjC35wvUgW3binRqgAk8NVjIACg8hm8BfwFy+UULUaE
ji2ctgMOwZSVMe5HqtT50/dvAowt7rDcs4Lstke8R0APLaAb0rzU0ZZS1eBXG09laEurea6lFtnh
zYpx3MTU6bGH3GychgyT+sydFk5w/c3j5mv/ZvN12/X6tqQF/zhwCJsXnbN4xP0vXpwbAlioFFMw
DvA4INinLqxaQZ4GYn/SfYalda/A/4YSSPVgoU9WYh0+wAWI6UbP2VCkNo3AagEdF206YI9gt/gG
rxhweZjXkbTFZ9RvtkFpkAkYlPs4qV+yfnf5y54IQQRMJ9clpgStbp/2oOPdDhLQKah7AaDLucKF
yc3Jblwu0xWhJuyPMnWA6lMmmSkXbfM9tfrCY6RnkWRauAfSnRON/wDI+gpYtH0KOgpk1UB0sHe6
xrtA4dMOWyTYatNZLUH7xoNtFnlq70n/Ytj8HwJo7857rlrd+OJfOvS8rdVRnkNh6fwz5W7l+O7r
Pq3veM1fHn79c5+49pBhwFAGzwjgqUJT3BHtsEAecY+qmZB7Fb6ONdZ666AlKpY5u6URJpyjZRyb
hWrTFjWmk5SpGkjkHou+TsD65ac0xVbh3IsQeXwzjBckqx5WjT0AM/UBcA4+LiSUP8pavKJsA95R
2X3J2nnemyIkyBKUzGoi2+YZAOGuzYIEFT9y0pp3Oxehp9Pzaht7AOG98Bhs+9Sodg5/T29HR1m5
2RTop9dNtDVeNBzVmFcrolgsRjsNytpcCcbDbcUEEAzLEYu1Ffwp6nHV7LZE7CPjQLElYaFgjznp
FfDp4ZsrXb39/VvvOHgYV88z4TYkweScf33vlas3/cuHVldfevHi/AkTFYQUzqCeYj+zwS2ctUSo
EG6eG2POIqDXuyFH3X4PRYglQwnaYLn9tOVb9zzCEmcbHuSMB344+kckyWhLX7y06fWAhHFWOdzS
gwKgXge5u0ZnOlr+3WneBigkiYn9Lx27ObHzIDNVFvWZ1TGf4s3dy0bmYg87JgYH1vadDpfniucM
zk7gWgzeFJcSHCSwzMJXkLV2LH4cCxly0ArKDEhHGwvFYu8WFtvg62KgFUwzUuNqwLg5YFw1YFnu
3tpuaAIkAMVHrx0P/fIfbrxqGw7ctiQzQ/G+j+gbfum1h/8Eo6pYxNf3E4Q+FW/BWbwCeNuzg4QC
aktLo7aj/EI/WJJYCecE3esNd85dN6BEg92Z4Sg+gNj5YYIow2gHk64gyy3IcgtYrtoz/AtbuTFX
UhZtbhonE0mlKaLQDo6j70cRYCX1+0qALTF2CUWzUfrgW7I9Yt7/FZ4Rk+rulRzo8q4JwpP4uc0U
IqqYmQs4L/o/anaHRIA/A8BHRBgG2biWKzJzzkp86kxxtj/UAdV2uCz666kMUS7Egj1R7qbLsHAd
SAeQZrWQJnuLtS2s7dnArhMOY23PZnowLsf86X1S0Xd/eHX1W949/sYRuHjrkk49CqxGHPi7f956
49XX4ZnnnC6nC5BzVLfQ0fkw5w1uQxM1LjvxfcMRCeebLjQ8aPMAXS2y0+Ix0WFhzFeqxDIO5tpF
0DP31YvXW5a0PH6gdRMQXC0xmSolH9IqxaG2ft9BzvNE3ztPgy1krTW8qX5+m9vCOy+G2Jxtwywo
mVm28MTvMgJCQE/WLNojBeqp23Z6wX3uQKnQE0QjpwXOfyWeK5GNmQOBnQdS6y5yZyAbBUZAN9vW
6BG+70XLakQ8ZGZT4jbm6Y3Fwd0THjkAdLzuzbnRxM9RMb+vu2ncePVfbPx3HKukUoOZnq6/Sa/Z
tSaXPO7By/u58rBb47sdk6OSkXxTiozIt3su5undt3JKlc0+O0moGfUrszbrj4CTWXk/oi7yDWhn
CvgDM4HORp02ayDmLbVpxyKAg2Xc+56USIKN/+PjFw+T1f6FoEjtT3S85Jlau1BOvtcBRwkYWh8j
lEEXJ/V35AbA2XipD4zKjoG48pCUVzSxHNQZJABOSImxd1mzcfG9MeHZTelP5OE/apbBLoYgd1kO
C7VgJbINzccX8s1gGUQtXReXbx44opF0KQFjrh/cI++vy28zvW9779bHvu8/Hf4aVWzMl7xt6fyz
5W6zs8CNTVz3+jdv/NG1n9Lr4hmJMP4+mrBdkQodFeOoucgBhEUeDekjKAPfzNOqGVT5lZWoUwb7
HL0t2/wTYNRDMpvgaoCY5xIWNxWzCUgbQLUpRwOV0d4lGXpRhF/gQJZuQhlz2V7JO8xrt2ZiJiGE
trhkqIhcHCvdToDq3F9uu/AopjhSx4DfA9Kt5pR9cb513qdYsfIA8vpYqOtoJN2k+IWG3ATmsqQr
yY1m3fe66Yk3inW8IFmOUBMNntqbeWRAW9Zc+LMePC0dMSxG2rY9xjS1bAR0PWD9cH70HiT3HZJT
1NE22NlmwXHT/uy3joIbb5bN//bGrd9fjbgZxyjpapj3KADg2hv1quVC7v64By3vDzcaAXiuoC4M
iZah3KQpvcK6BWz4k4LSdEGjTIkvFtcbBsqZDyAZDPnTVNSwmhqD7/n5bdzRwJAKK/6ouiu8ezAB
AGRV3Tsx+up8udNQIXZG20I0Z11JF+I+NYecOaSXNjHcxFMfgwAa/xHWyQeNiFMHavsOe86GbSUD
H4E+4PXzPdTrRXloyVjdY5VaZ3iOQ9DHFjZ2yEbnSYZYkLoPCRS3K4PGSkZIc3g2MfCdJJuF53eC
eL5+YNjb6JXFv/PXooylTX3b+7Y+8V2/cOBrjvqQmpl0wZnD3bZ9u/HGFm744zdt/OG3PmP3k889
HWc5lY1GzXmY+n50hMfQJ7ba7ULreZx9YvO53FhUHXDeGcn7RWOSrCkjwVVSBHgdKwDDANGxCddC
EU+YEtmtVhOOqHxsezCMEBXYUXPZJW7PwaH0IybU2RjPoaPv1J+5sIFbVp6fQ+qBKHPuu7enldKa
l3TaP9WE22NKynQiAap0dpK6izL/GoeJUvB3kq9YFZrJ0+5LBWOic7sl4rym2R33FMTLwrAorRM/
j1OU2zyfcdWs/RCequTgBcL6bucZdk1obINfppwC3Hhg3PjNNxz67c0VrsOxTKLbexQAcO0NesXh
DT3nSx62/jCpW+tcP21HWPQ2C6dUR0eo5ayKS0qtxfVe3HqRa5zzsyzQyJDwHCKYRNdhipIBL9t0
NQ5tI5Y92+EbtEIJ/P0YvcX1epyAAA6pVpMDO/1UoGNQ4abRHmDWxRNipcH7HQLFUtQl4bor6WG1
Q8vIUs5OG2b+HMQ0v1eeJG8iSE7eQ/Aama/wvwNbsg2VndRu+D3hDVkefpGyoBkPmzZIiUtkW24c
JMYUIYcRvPQx8/gGYFMmezYHgAzS7lNXhN3HogjWSH97gO1Ghr75PZsfeeErDz5bFUf16sDt0vk7
eRQAsBpx8PVv2fj9d12x6yn3u9vinjwgqp2fwOYzMqEM2LZ5/Gfk75TJyrLOTRN7IWYRzFMRl0hJ
J4RMA3nEFqhN3IY4wHS+klv02hlXtGlXU8A8oOstZofCSabCQUmhux5q22Bh5ih9R/KOjuKBMf+t
n+Rgl2bz+RDsmNzDdANSy1Q/q1TF3dQEs0JPBxIuLpOkwbtl+wAAIABJREFUBKzh/bbMSm8hb0vf
Y8SBGq0WC7EBzrYjwxTfJTG9yZ4BjvVQ1lcYLNbgcYVxqyHVYDGQhjv5vJMESLnwu1AQ12zj4cdv
HA+94ncP/dw4HrspB6cdPQoAuP4mfPia68YTnvnY9ccsBizCLXZ0PhI4oPtNStcX94vhZZC70QsU
QDpgI+SBxrAW9hsDyh4I3/nGllqoLFsmcbOj/meWEBm44eCsNx5KJTmuk5hDJ+DifduGj2UK3v1l
ncSH4nrYZwTZMl+5H836dGNCYbbZDUnuYJwbNL7Xt1WqDSUvy8bUWfZgPQ6Rr1Oo44+eBwvFsNSm
uOsjhrUVhuUKsmygEd6EKynxNzGXBo9477GwAoDUvLrXt2j5ZGjKNG4NWNn0pMkn7CyLjr8yU6kA
o2L8s7/feOf/8+pDz8PtkM4/c7jbEYECAD7+Kb36rmcM933AxYu7Fd3s/kCfk1QyZDDQ5akABwUL
J6lToBSK3Nbq11JhAInH380VGNKtbPNGxANl6OrJ4FjXnXCv7b6NptZcVN9MhyaKjunqij+bNgcM
/rSsoD0kZIIWm3bKIEmVcUHVo5y3wNVOqfrZ0aXxkC5HdYJMsSWfm4hgI13zxsUJ9LwUDNQeKSlP
gIvxRoYRslQM6ysMu1ftc22FwV6QE2CggJ/iFtMI1GZK273XK5VHvjEQKhY7YhrRZHGhWKyPDbCW
5lFETIMPOep47zJiLHr/NVs3fuNP3/JNN9ysV04H6DNPR5x6eLrhZv3g/2nvy6Muq6o7f/ve976v
iiqqKKAoRIpRZBAEIpISQXAAVKKx1SSrVwxtVrPSWW1cS2PS6Ra1bVTsaCtJo7ZtWgigHZIWlwY1
MkkRjARBkKEKKKYqpqqigJq+6Q337P7jnD2cc1+hIjUA31n11Xv3vnvPuPdvD2efcy789vQlZ57Y
fc3eC2kvYZIW3SS11gOffKrvRnecN+YzldsxFmN0GYScAgvKHAUSpcNSNQHiJO0rlw0DCE4bqJIa
7eqEXB2G/E75M3ItA8pO5y/kjjUvSSQqukQI0ZvEsZ0eMa3zdeMeF/VGzvmbZQLkoMwuPzdOCs65
9QBtVvo0zcL1C+xdeSYrD6OuRfO0jtQVkul3dXRXAVUt5TGoG1B1A6gTILu36/Sq83WZKRPvyb4g
YgZEumHr66QFlOOcHeLktE4/JZuTdAJ1WJxGjCeIYKHhBU2FipP2oXhjQYLTgzC4/Pred1avCzdi
O6ZfSqMAgHUb+f6NW3nPs5Z1l+lsjiTpI3XVm2Qok7Y1EQuJGi9aRAE2QqAjPcKO8SVPL3FF9c7C
j4khu9REvGCjfCfG4+MiOSyi0mhUGI6Kysr7RkzR4YtsHYBNKOb9krXNtcOrtfasxXFIH2Rni5E7
19LXCzBmaxVapjZSl3e8j6CVgwJb/p5vvYOiNqB7UHEeddHeqk5APd6gGo/aQj3eoJ7boB4foh6P
JkVVp+FyWoPfHCixdNLG2IXvy3Xqe9VCOZoqHQa6nAAp3asR4yrE91GzOUhJSkr96JyznBqqW+h1
Q9Q2ao4xEzMdcK8CDRMtRI2If3z34OFzvjj1TmYMRgze85J+adMDAJgxfHRDeOyIA+rjDl9aHaAI
DBhMtmwFyv+o/avmDxVO+psOpp8xyMRT+iCRPcj+z64tJDRXc500yRssS9Ap4wQDDFcXJ5VMLXbt
ck23eBNpr49BaGsYWZWKtpX9IXXLzAAHUBYR6+rsuVzey4xjyk0PlrEZBSG+nvZFNLhcu4LWibVM
B2JSzcKRkiasFSDqOUNU4w1IztvsJC2ia8yqM0XqI2IF38oFSVEdJTvVDOqE6MvoJEavrQ8yLY8S
kKQ/mzGJf1WHdVpUwUcFwIhZlYSUQqNhWKGZ6YAHBAwITa9G06vBgxqr1/LW939h6zlPbuKV2I7p
lzY9JG3YxCs/c9nU/1p21IIjF++BvUrCHbndl1M9yV+ruured7HF2ToRLvLx+i3Zb9uSbHGfiMSt
yatMTBbXIPpwxjQFG3hVU7/GglkbT8pEEhougEe+PVq/YibFyZsSvPTrqDaOmCViIO0HZL+Jml1u
5jNq2oDY184AojQv9RmnkZO7ltgLHaaKgU5kEh6mCEsFFNeI9ILsehYo+TKqgLqTVnB2ZAEf1Ewo
oyE1VQAhmSBcgWSv2cJjm4kb0TxJNJgKHFg30lVNoIF1igdBL4uqFOZdNekYwyq9wgrgjsR1vCpx
eqYoTO5X4IYwOYnBxVdN/78VjzQ/aA3e853oVzA9JK19hu997Mmw2ztfP/b6qkqarnQk0GLe7Hsp
xUuZlFR8AyDn5ErvE0s8hN33l/KqMKhUyaL6KphZQPCzGZyu4yErFglodYKLQk0FZVRRNsnuCQFq
XV0faG6ZRmF1z5720YEjkoG1tDd+b8UwpAhLcn2UN5Sy775qPupW3vXD7GezJNaAGUDN6MwborNo
gM7ufVRdBkIMP9aw+lJaU4yireu4d0M9Hv+qsQDqigZg1YnNI5vxkQ2LZFEekWoPUZOQlcKRVdm1
0jO4RvZWFGfQOqaJVLX3IdhsmJqucB2C9IxEeooTXU8gsz/RbKgbQOMB1dwh6nlDVHOH4dqVUys+
+LXJ34YF12+3tHRf+uVND5fC6vXN/Qt2o8NOPKJzuAYvczbGmsp7qoa34BMZoVJ+qe8aZliBNLLA
3Covmd1OFXPrHArRktWd4aIkXfRZqz2Uv0fie6FcskpbqGiXT2R1zzq49WTSbEqPPKf1CiMAogxv
zqcZyYKUEkiYhqctN0dgCxx1kOId2as0EEIjNjYAVOA+oelVbj1D7KFoKZKzGJPGpnuIit8o9Ssh
AgrFwEeQ5RfV9g5Cv4NmUAPDWpEx0mMxAOLErByoq9Qn94xjbh3LWM8KLthvJLRbfE7WdRhBzwSg
4rgGqWbcuWaw4T3/bfKd0z2sb2W7HdLSfelXMz0kTUzjsb++YuaCZUd2jzjh8M7h5okmqN8CJl1y
Fdrkkdqs6REyqrX3HHPIkl6953/nGOGmFoKqvjB12QE83HNW2aKhTt22sSP3nFQU7lcJrbX6MihK
iOJcSuuRESCRZcqOUEc+YNOHIr1VY0rVdH6KnL8d40uN5H2Pm2kwyY1NBhbeacGpTRR0cRQPKMq+
XoXhYAy0OU4ahoZAw1Rv0SWrNNOUUUcnHyrnYBRGQp00g475JgwcAR5WGM5Ekq/qNCU5lvsz4g5S
gO7jqm0SJ6jUqLKZEa0lFe9YB+azH0aAgmn6jPskGz59bu1TYfIjX57+6DNbtq9fIkv8HEwPSZsm
sOauh4Zb3nXy+BvnzqE5MUOVfS5YipVBFSk5MVJCWvYPOL11lG+USDJHLrjInvHX5J7xjsORwJC9
I1LC2iS1VsbkvALqVHVzmCqxHJVrVSkrdnSi/DOVpPnrXdEgFCDIAIJdu9nqwEX55Ig5ahOubr7+
nPdJ/J7UffaRnSkWIJjJVUm/NymcOVjddIZG6uHPVU0rRHlA4EEVP/sVuBe1EvQqhOkaYSb9TXUQ
prpopmo0053oEOzXccKrATCM74fpDpqpLniqjt97NUK/Rhh0Ywg/on5g2w9EIqwIkHBtAtkyd1lp
q6CQJ1ZNQjSQYqw9LUt/pK9TPe5/4e97l1529fBT2IFp6ZLnZnpoeuIpXnHfY03n3SePn9ypqAac
huB5SDSOAkENLGz9gqhvGjqdgKbynei5XaYuWz3sLlOFWqDirok4Db4jaMBUW1WBnT8mAxGXIdlH
uzLtR3+RRqE6WKEOCeHaKWoGEDqt657zM0cEUeMl71Ql/07ShrymJ4BrWlYZom2dTu7T1HjrR32D
YJGQgl6yjYCeiJb6y88wyDXcdwEaijWjbOysfNGQwIjKQ0PAoAL3avB0jZCAA4HsxC83qNr3qV9D
U2Ew2UV/6zh4WNvSc9dU8Z/k8sWjNSVatL6UsW8Cmr+/rv+TP/tK73dRktx2Tr82UAAIDz0R7t64
lZecccLYsQqW7BrrGEYlWAIAi5TEL/zT6TPPUayZufKKqE7SR+x9DxYKTHm28mNh/cAzAAAHGuk/
vbZOEOlsOqSVldV1GymLyZC3nKmRmxkFWIyAIBkjILWdSf80TzhVWpgiK7fILwG+6dFSDudNZjit
J+WT9hoxLQi68lL2txAfRHTyJVCo3dimjlSS0Ha4P7a8JG7BSMqBn1g0DIRBnIqsJIIXHMGDbX8p
FTQ1o57ToB6LcRWtri/Iz9OR/U9ufASMwNfcMrz3335q+i2BMdMa0O2cng+gQGBMr1zT3Dk+hsOX
HdV9BRknQXo+syg8mo5K0pnFZ4kRuS3oXie0dWqI+zEtpnIg4SVzVo3C6y/3bMMRV+Hcza/3zfqy
dR8t2799q9UhtsM0OQ0Kulu3Be0gMbS02frH+2so1Y5GAEm2XJ5dv4hUdn3iu1Fy1e2U1Z6XPJJ/
Q4+AhDM97f1YVNG/CSR8EJSPR7CZBht/TgCkdfYy2Plr9NMDV6qKztg0lGZmCM2wRjOooWs1pJZp
JqOSACt9H2gJzWTGRG1NNBU/DgZsgcE/vad5/F0fmzxzpo912AnpeQEKIO5dceeDw/sOfVl9whFL
6/0E1cXcAKwf4ri6CTYPCCWIlH0nXzxjKpcZUXtgyafc4sD47DM2cd5O8/QLAZF99/XQauTXxpSu
BE+sjjaeXaOg1v8xLwME/6wyOPIuI8cAvoP822o2OjPLgAHG3CMm5CiBhIGTk4wKMmRMLGXBaYCu
P4igQUsaISmA4Z41kHU3AOg0t2wcE1x7WdpDuiNWdm6r1iHRKQM8iIFPAKE7ZxhjOLLRJYew8uka
6sZD6wCCra9h7Tf55EBY8XB46t2f2Pqepzbzne1e3zHpeQMKAJicwdqb7xmsOu4VnZMOWlLvJR7b
lsVQfAeQbx5qvejulT3twKLMzF0r32u57L63k9qyTu3zg6pqsXs7Nz3gpL5Vbls4oH00+teM4Vny
zpqbg5OW58DAwIiTH8a0PLUSKNcOIlhQNgzK3Iom7eqWl/k0pmkrUseqzgPOFDCq/LOMZvTjLns6
276mNm5Ke5WX/vFm63zdxLTyXia0kikEBtBhUFrEhaIuotVlAOm7J5VjTnVBcOsnM1MJDzzRbPqd
T06c/dBavh47MT3XOIptps2TWH3Tyv7qZUd1T3nZntVCUQmNd50MVJe6J/70eyFZvGTLPcU2yGrm
uPqUA68SqTJnqQQwReZAMjncximOEkoJnhG+1MVFlyodOPCx0krV3Sdrr/KjON702pWZXkm8ZYwi
fw44AN8i0vfUaZkJQUEStOuqzDEqRkBeozhlOcZAl1OfOeAIleU1qjOoKI/si68zcRzXKoVEVxUr
wEQHtXG9smXKTx3Y/mBtR3/S/vgwQGNANS5AgcxJr9X3mMQy/Dnwat8reKR7qazV68LW931m8gN3
PBi+vY3u3WHpeQcKAHhmK1b9ZMXgkdOOHTt18R7VfE+MUfVHNuAFL9otN0itaSSMBoH4epolSfPs
aicCjvklDygDmoe9yvY41KoqMZAOfnbPcrWv6iA0KVK0AgoYon5l5gw0iKrNR5RrNwUgtRO3+irP
De1OLvKRvshMSiQmcKAt11UnHnjT2bOHzsIhqrHIXJz2XnBdoN9d1q1y8uaQ+0xll5zuHlPHMuXt
eDaT1/o+tikwIYQq+iLG4tmjftpUvut7HMPOLUPLj+Ccx/pE/P2xDWHr2edPfPCmlc3zcy7Hr5m2
C1AAwIbNfM8Nd/RXn3z02GlLFlXzhB99Z2SbqijHFaibBpE5H1T/u6CFagdKLwbzdqq4G9CkToqD
Mj4n+VnIrquhSXVHCFmrRLqpVmpMTCMI3ujSxZQoo1OmGbRUfXbS1H0fDRJ2fwTr23enUWS/E1rZ
Zpd+JqQAkMCVjrVsyMJNhdCvDOh9Ee5atRyPHsivqbjnz/YIYlpI+UojfhYFaYqDigxHdUPsIyaK
IeR10HIiTXsaFJpiNYcklxEGlwq0x58OW9//2YkP33Bnc/GImuyUtN2AAgA2bOaVN9zRf/h1r+qe
umTPen7m9slGV24Qytvt5+26VA+35Ri0OABX/giiU9qTqbgWeDlGymZERjOyEFUWgyDML5qGElDx
vcyPTUobQLk8i+Y8O1i4P23LNp7f1v2UlYCjOlY980qHBoqBUFs7aNIf9yrZIiQ+L/3k+lLvaffY
DAklZtU+yAA9RzUBIzk4WE4Vr3RXbY8RlBaPiWRyzRWao7QRDUWNSdd4JI1U8yF7keTTlzNijB58
Imx6zycmzvnXlc03t93xOz5tV6AAgKe28D3X/qy/4rhDOycfsKTeg0pCBRC77Vnui8dP78akmjoX
n/DZeGZAWyq3xsqHSOd1UU2iZHAU36XeSaMoF1A9C+tZw4AcHDwDZiZQASrp8ZEluXwzgGhpKinr
EiRGPScmWwmSUg+K57aA00yJmxo1882cit7nEnEmV929c5TcM5FPC1MUafFWYvoMvMjlQch/T9ys
EZTuTxeTATpbUnVZNYu4Iz2htV1ABjh52TKCK9c0T7/z3Infu2dN+B52sbTdgQIANk3ggatv6d++
dJ9q2asO7Oyd/cilGoZE6DkTko0ilAJbzkaK8f+VmR82JVchXzAlDI0ivwwarGbyTpA6C9M6M8Y3
gZ0wLnwOLYYrk+8Tdq3zHv2SOZ3k1Tq0gNIBC+iXqkerDPmTg39KAC4dm4LmUqq+QxlASL1sZsUD
a64j2EFduVwuTRh/LU3OzRfKAM2P/si8PElyFY8TrCnupNWRuVVClXbTkvZzSbtWDIB4ftZt9w/X
vv0vJt7x6JP8Y+yCaYcABQBMzGDN1bcObuzWOGbZkWNLCUSl5DBJ7Al6tHRUwZDd4NZgswMFSDmZ
ep+e9ZmxzdZ6vwQy4oUySAZqAjjKpEXtVTPwq1VT2wuAgK9rqw4jesV1SuZc3JY2NQIodNds947f
CYpD9DnIfSp+13o7x6GWVZgI3scibVPoLs01uNkz4TWfly/HAQJpM0xjyGGnADayckpwAMVzU0JV
oZrXYHyvHsYXDlCNNUWdSP7pIVK6pFzMH6sb33jn8OG3/afJMzZu3XlxEr8o7TCgAID+EOv/+c7B
NRs2h/1OP757VF2JEoY8fFiTh3AjwhxMyjgHe56D5Rl/dqqpnKVA7RL9TIXk5bcsAzA6IKnIqcUX
2a/bYPRUvpewBkpl/7Sqn0nnloORR/AqO3BLbWG570Oqg9ubg4vfYXl7QCp91VpnxyiJ+9qAUQKb
w2F3aX8OiLww8FsHmAVDWR5ZxipQMryJOMgVAhG6c4foLOwD4w24NkeKj1FRoCAkZynFv5qiA7UG
hgHh8h/1/vU9H596U2+AR7ELpx0KFADQBGy5ddXwmjsfHnbeeuLYa+eMobYBpxHqdQ4EJnNKuWqE
YfYuil+Rh/3q++kbE0i346e0AxHBM6kPSRY/hCRufZHrHNha7J3Ucy7yzJmlDRJ5u+1IPXaf2d6Q
bfZq5aiuDucLkLLIaWw+NiTb1Sy955e8A7n0z5hXm5EWSumGnzIGqR2EFPBgeXEAOFQIcrao5Ch1
8vQjM1gk7WJ4QItVLUBCyhIM7QL1/AZji2ZQzRuAxhqgSrEUWV9KjIcbI8GTZKFMTPHg05dMX/an
X+r9G2ZMtQZiF0s7HChS6t33WFh+/e29jW84ZuyUvRZUY96W9STs72QLltjmoluMmM2gUEakJrXd
LY57cFPyQWQnfGUg0VaNc3AQoiDEXZWsXuXuUm0WdaCT8gqunWUshdZPNoZlp33431N+5LQJBYOy
T8gkvbfd5VM1BAZYwqOZ0tmj0pfGFHaAdA7kmRmUwKFKZ23IIcDwwCCLwDJmN81Hul6XfSv6pCLS
GSZSdyrG0rddGRr2qUNSA525DTrzB+DxBiyrQykdZl2z26UqlhtlTQK79PfIkzzxvvMmP3rJVYP/
4npil047CygAIDzxNN/yTz/tPfrKl3dOe8XL67ml2ueTl2ye2VXd9T+nEW9NlxaAkSO+Awmpg1eH
RYq6ZOsVHCD4uAxvu4Ocqi6fcM/6igmzeXFmv8sCMyK30CyrWL7gSt4TDUMBR/pH/ivte9c9hiAe
LFzuybZnkrKQlQFYXyqz6wrQNJNQI4VsC2JxC7Sypsr9BDS6LiTdU4BIU6AC+nqquW+a1K1sslkW
cQZnLKCaE4C6SdmR1gE1p414m7iPZ6cBddMZIt0GVAW+ddVg3ZkfnvzdOx4Mf4cXUNqZQAEA2DiB
O//plv6jC+fSaccf2tmtSptAjKBbd98TuqMiLyrJBthjSyFEjJUIcB6TEdOsgKrHrbyMuXPnoTcn
yum/sgAHLGpfO6DyHVECg68DKDMF/D4Mak5Q7MPcUwdlfoaLiUhAl7uJtTvi24lBxQOcgURh6ilT
SdSs7Cnh+1mAEgJA6nwwsE2FV7I3hYBDVA9b5XrA9PSge560nBKur9nyCU0sP8ZPkD2a/A8RTdL3
GkBN4Iow06C56HuD5b/zF703b57EilEjuCun/fdbuHOBAgBm+rjrmtsGTxE1bzz64O743PHS4mt/
t/iE4hkhDGFMtdXlx1yaAkKkpBdi7qsTirKf9WX9nwvGUgYvHH/pHRbik7p4s8mZNKUGM0pz0Cqk
vLKdy7XFdkFtzrG8xEb3gCeZjdq+VfhVCilNK799vTzjj+orDyBRrQym+bjVmxp/QSPKHlUvvTQN
y9513A9HbzJepU9HMmRCGNa2Mc1YA12kphTB2m0E4LF1PPH+j/U++VffaP4ovAD8EaPSAQe/cucD
BQAExl3L72yqPec3pxx7yHg91rVhGiVIaeSnI2xnlwOmIup3OCKrRHoBGrqdzn30XmxQOTOjGfqC
HJGZeePozMoqW+UEpzhzBezKZ3KxnYNEBoIj+srTfeWysXqR8XbBhGSNci8ZY2sVVMtwfgXRDhyT
e78HAFvqzZRrBcqvZGdoSPezew/m8BYfhvmdXFsdY2tr2D0fYvQl7R7QXdxHtbCJi9o6HE2mDqNK
S81l+AkETs4iAhAC+Lqbm/vf/Ef9s+5+gL+FF3BaesAhz21z3e2QhgAuW/Mkv37OGN4KwDZrGUGs
pYaY/eqkqtG0EGjImFadW6mMaPcn9paCAtI5DIBu9ursXClIJFb0qEuGsSGZyfJsiV37yIOdI3Q2
jYPdeyjbBLun5scI6SvdI8/KxjMlIHFiRukX2STZ/DoJFMT3QCZZE9c6kJDfom4oKyuzYxtLm1Mr
qhUwMMjax7Z2x/dR+s20w5iXaBs6ZKLpdIBqfoNqjz6oalDtTggDivt8VtDVsNqlzLHvGHhmM/rn
/c3gmxf+XfPHAPp4EaRdBSgAYO2WSazpVIA485ScRghVAKqKeqb1woIIcYtzOQ1KlY70IrntzNjK
igTIrdgOFodl2sPRE2rrWeQg1Aa10YndCx4IPC75fEf1i3RBCxjkhxFIKwzN6T3dWV2YMI1LhiNu
V/Hy+D0i6F6X0VfDeSUJIOZ8JiRpE8SIh+pQktIJhNW0UkBPgOA0P7+DtZbjwQhmuAq4+XUjnKrZ
VBSPGW0Qd/TuMKgb149l3Sn1JkbTICy/tXnw9z82/Pfrn+Htehbojk1h1wGK8S72OXBJZ38S6cwF
4VNOZxb0lDOXfsaNGeJioFr2DnCjTKxaC7vCRNNlEc+JrGQnaJLNXospSKDFB6aKu8eEIX3bfOhy
+Z5KuqzxBUP4AnwahU4jwYH0WghfGIZ8/dyL5alwRMjNAt/VELBAjFkmx/gjtAfVQkK6IcMrFUoo
qNoJOfoowFH6UBb5yTYE+eCwPccROCpicL8G+l2gE4B6kMwLUVDzDnhiA0995ILBhZdfHT6K0V6d
F3Dq7zpAsWg+HXjcoZ2DdYMPrzMSg93IFvyuD0uMlBJQZSChy85FCyEh6BhMBDhC5KiCB3mQkZxp
MgUKYxZHxCT1VdazpEzo1Fy7J9c2v6BZK/Fan7hqZvETGb9laOM+y6QZ2V95nCP7Riapz7LuxaFJ
dByKKgKNuvQRpiFJdC08aRWjhIK/qRoSOW3KdZ7vS6eLqkNU6QqjtQ44wUMUImgPAO4DPDfAj0kk
iUgHMwMO37p2+LM/Pn949tQM7h3Rwy/4xMOndx2g2HcRHXr8Id39DCTSFCGcCZBMBY8hrFNiyREp
DrR0HmW2dZqf+4cRr+Vptrg+l0BCHqqSZGLNxUkyp2X4eCPZOcthWs67rkGel5nQkk3mI8jzN93I
AR9g+1u6AuXx0oZXSVtIeQNSshWgfuNaAFwREFhnitoa3AicEr/FiH5RUAAsFgK+r609/v1S0pum
qF4VR0/uo1D7KgA8IDSDSs0NQtSGCIzA4NtX8bpzzuuf+/P7+OKyaS+q1Ht81wGKl+1ZHbx0cb27
HvXHkUDitRPdPslUW91EJ5o7vVpmMACkwa9tik3uAWp2UBBVOFInO8aPkttChU3yOGkoHO+klDrt
FCRMX1DilLx8VYV72PtMrN7kmVz5wvQRLyGznmP72YONJFHPFUTYOflCrCSXuj0c44pPwm8Gk/oh
TWRkGoHVIzHgCJCtWvPEBWOPIAt1LDv7Sfwk4riMYOTGy2lOsV5OFHAFUJrhIOZH1/HEn31xcOG3
rg2fBDBo1+DFlRidXQMo6gpLDt238+pujVoIxyLyEsWTU0kRiUlAgmqk4+5T/H3FOYGHCjqPllY/
Ak5welVUdlQRXb9O79YBQDxJWl5uGxgwKZWAKluRCQLZ9le2dkTaVTEwxqjH06nbM4TQqxSkYvaS
J2cSleGlqUMgxHJEEkJzcY/4Tk2mA6cXyXW6mlxShMQ5+Dykb4RHyY2VKYZZkdouNUuS5AfUaezr
LVqG3hXNA25MZHyclsSegNKz2ifstQ7LlwTwGHhmC/qfu2T47S9eNvxgE/AUXiKJxl++awDFkj3o
2De8uns8qhQXKMSvQIFc0sn9tBKP6wCqZZFOkgQom1roAAAOR0lEQVRCUUFOs6a0H2Yq1GXrp+eE
cR29ZSJMCNn7UrxZru8UIBJpzZYayz0gauzoMKo5DaoFfdBuDWimg6Y/hmFTg8CoK9ayfBularlP
IjpqzddAcdanQjqjQtoqz4jmxEkdsHZKg3QGQ8ywoo80L3fftb5gQSjgEJLATsgVx5e1TG2HagHS
IiurNf3s1SgPbh4khD7K6SP3ThgQeLKLiUEYXHxdf/m5X5v58HTvhRdZ+esm6izeNYDioCXV0Sce
1Xm5bKmeebCVG4rDfIvDYAACQo3ccCbIsffcOOIwhDBQSqsRlQ+9E7OpwEOy4+08kQloSSoklr+m
bP83qYv5VKjDqGpE23gqztvXshFKS2qOKNPd9k/IdKNipGOeKLXZtBbtOgkU48iwDeX+kgIQuLwv
IMLQPUSL7tDnKsqnL8X/EZnb1Qm2/sPztwoPKaPQtGQsdIpW2h/IQF7qWzG4S6A5AdPNYHDx1VPX
f/Ibk386McMvOYCwVO0SQLHwFfvT8S9fTGPiNTKGgDGSIwRywTsAov0ZKufUc0kkvmcmGBhpgfIT
IwVYkR1rl/7yNR2+DC/SrU6tioxKgoMDQtjaAU9X6fTvCmgIlfRJU6w6HSEIt5UYAJoqxiakyEKk
8zbFL2PxDOxeigUpUABqUo1smgBeOQSlhHdjYH4kASvkY+YdngICZdtd9+eHC+V1VEdvohObXeEE
YgR0GdPjM4O/WT5x3Xlf7334xTqT8asl3vlAsXAenfCaw8aWRRvWSRZRr0cQWBxVjpKHkOxYt66D
9CkVTZH+klQh5JpBqNz5lyLVKJ24TZHJ/POwOpnaHQtl1Ylt1aJ33lHrfZfvkMCD2iSpT97J6vrF
+Jk1hkG3wpcZIE6gAMQ1CjUjDGObCUgufs4CoIRpWZy8vrBi6kWZtDjFi0i0QAOieNYp2pyswFCA
BeWrSEvAVb8J5Yqi9jiz0QWEBuBM2/jXgLBhMkx/9fubv3/BlZPnzvSxCrNJ004HisP3p+PPOGHs
ALlWnwQXJNES0IkIvF7u83DvyJy3xDFE52ZSO0VaNtBYCRK1VMBBAUTy9/WIsQFOP4bABzs1hjw1
O0ZwurH+UXZNZobBaVX+Ve8oTYFmcUMX50FMnBJ6VWb9RLOH3SpTTueDIgKlPmvaXmRqPxHJ2RAo
QEv+6jOpEJh0hzGSWRTRGCsAHOKYuPGzdThWXKmfaX8wHKkYLW1Lv2sY/MC65ulPXLH5a/942/Rn
mTGB2VQk2ulAsc8xB1fLDtmvGms5pSgSZ8seTZ+51PC/sxEe0joL3QG6TiYE0lQeISQnnWgPChDw
0j/XZwNsgRMQYyu8xNL6i8YgDMGWl+adgUU+/ZoV64mdYdOUwiG+Y3xYtJSH4rmMc1zmrrMZcBqG
aEQu9M0NljkHPXzEyrkatDShsq3SB23TyrQCH1cxqgk+u3jfZlRkJqQ35HD9yt595/7DpvNXPj78
RpndbMrTTgWKRfPpdb/xivHX1RY1lBGy2aP2W2BTHaNk4VzcioQUKZSYnjna/DwkbNoKfnR9GO63
qO7sOa8iBABDQgjIgAIkRCl5xJsi4WXxk5dulDhd95lMOi65OUH1B2jIJdrahO+HrL2uO5K5ppu9
uKCubHXlqBRcGeza52Y3NPiMTaPQIfIALv3kFoN5O4sAcJq1kcaUOMWw+vs2SvVo1HtZFiRZaZ4e
8ipiNAx+fGOz9cKrJ678P8snzp/q8cptZDebirQzgYKOPpBec/rxY/uq/V85db1y2547aalE6lVw
dppH5QhWVPwAzEyDb7lnuPW/Xz551VW3Dz7PjNvqCse+4cixD33orPlnnnrU+N5zO1Rl5aUyxYvO
ztZmRE3FTjxjx9vUZniprNjoDDVzAKRZl8iclTu9W6rj+d1Bl8psldCOuVvA4zPKNBAkf0TsxFgX
ed8bH5xAAQoO/rr1BxuHcmepzJTw92TYyBytcQzY2pGQIwcNm4DVxxJgTvd58L2fz9z+6e9u+cJ9
a4f/gNn0K6edBhREOGbZEZ3TD9m3rkRFp4ZjKHAUQdCgh3JWQcWOEJ+bOmWoczM0wKPruPelK3o/
+er3ehdMzvCVvg5NwG3Xr+ifff2KZ6hb47BTjhz7jx85a/d3v+HIsf3GOxT36hCQkD8vrn19RNMQ
c8ep1FndMkNb/CTQmQVxIEqkqdRBZghUQyiqQJwHD3lNRaY546Xd5wSC2bSoqveSr3S3ITMnMM5m
INyYmIqQMiryzPqtFAap7ygtz1eQQTQTGUCnE1BVeTExf9Y6DIah+dnDg0f+x/e3fuOHd81c0ARs
xGx6zmmnAcVh+1UnnXHC3GNUyvofhbCFJ5OKbiqpV9eTlBfODMDEDIcf3DxYc97fznz5nkfClwHM
/ILq8KDBqh/d3f/Qj+5++kPdGvu//ojxP/zzd8x/3xuOGDt4vEvdGDbNym4Sut1aRekZWXjLg4bG
IiQuY5N8VTfY7IlIeIkVSRvBVAR1Noo3n9VJSg4QUh19UAEEbEj7UTQHizFJkj4xpVbXTStk8Ss+
zLKl+yDTIvy0KpdfMu0Lsd1Sn/SQbn3XZe3LSvqKCf0hmtvX9B++4AcT//f7t09/ZdBgPWbT85J2
FlDsteyI6g0nH92dB6PZRExAJGxWE8MLnRaBJeN12BA/9EQz8/nLp6+67Nr+ecMGtz/Xyg0aPLZ8
Re9Ty1f0PlVX2POYpd13/cmZ88456zfmHLvnvGqu1rIECakPuWlYR/0WM0BK6Kriu70kZWWmN7Wi
7wUalRqEyZHvNK7xG+5Us6yD0zVL/bndDm2MgImMje7tkTO9jlMlm/SwtdEpGMUr206lloGUpyxP
b0jUO0z2MPiXVf37L7xq4pvXrZj538OAp58t69n0XNJOiqPYZyG96bTj5ry5242VEK0iauaOO1js
WLM9IZI9zeVPTCP88ObB4x+/aOrLqx4L/xPA9PNZ1ybgmZ+vGVx0ztc2XQRgbJ8F1YnvP3W3P/mD
U3Z746FLOnvXNSq3kYGCBbMscXYiVT5kCtKQMdJ9Q06DShUQG79BzF+WrTRQgNE+cwvbyL+rzlNY
HzN5qwDawYkHQRxXgjLnWpLaUNYmBtymtslvEjieJh6M222ihhWIRpoj6ZlsaXgqrAkIjz3TPP3t
W6f/5aLlkxevWjv8fuqN2bTd0qgjJrZ/mvPO19Wfv+jPF35gwTyy/V513t0ToTm8RDNlACEAj29A
/6+vmP7xV77bO683wA07tgkxjXWw72sOHnvvvzttt98789Xjx7xsj3pBRSABiZY6DRjh6+5PjpHF
DEkmBVUAjQVQV5jPPxfBiTouqCoQkELVVXlBDhRi1jn8TRLfqW/6QekYQQEPc7SKs7U0NswcSe1z
JlJ7UxkJpnLTx65OsZ/AGyfDxD/f27/368snv7X8nt4l/SHPmhQ7MJ100rLTd7hGscd8OvX0E8bP
WjBfjYyMl5QBRO110my6z3zzPcNNH/v69KU/WdF8Gti5K/j6Q6y76f7+l266v/8lABjvYOmyV47/
wftOnvvbb37V+OEvW1TvXskujMSZryJjDsR2c9ISEmSqH8DCuBHXnDDp2hAijv6EIblYB1dJN0sD
ThvqimbQUu31IlVZHLOm0WWg4J71flQSx4zsYh68SpJnwPkXNA03j28MT19318wdl980/d2bHuh/
qzeYBYadnXY0UHRPObp689lnjB+oJKPMY3a9j2AMBDyzhcM3r5tZ9alLZ/5y4wRfil10q7HeEI/e
sLJ3/g0re+cDQFVhjwP3rt9y1vFz3/v24+a89viDuvstmleNU5XCpYSZgrMLVBsRzoEBiixMY5jG
0hDCkJLZYsvrJZXM7X0821Qn5SFdjOvCw93UbbkKVsoXLUOnRsvoWal6QPP0RLP5jkcGD19zV++W
H94xc+UD64fXMr84NqR9MaUdanos3oPO/Nx/mPvVs88YP0g3qCkdd4mamgCsfrLpffbyqesuu6b/
8Sbgth1Z1+2UqFNhn4P2qU9/06vmvOXUo8aPO/bA7tL9F9UL5tTUEdOiSg5NBqkqn21kC0SNok47
fkkkaVIl7Di/onRnDYnaX0xS2KNZdCfMdBDTwgGF+S0yFSW+FyIgTEzz9Jqnmg23r+k/eO3dvRtv
uKd35ZNbmufscJ5NOy6ddNKy03ckUMx97ymdz1360QUfGOsITZMu9RVR2Osz33L/YNN//vrkRTff
O/wM8JKY/+50auy//6LOKccd2H3tiYeOHXnc0u4BhyzuLF68oJ43d4y6VX5CYPqMzMnpewxwTaAB
8yHoO05RyTSKzPZzH5xrAYADDPdOAId+g8HETJhatzlsum/t4NFbH+6vuPmB/k13Pza4ccs0r/51
Omc27dy0Q30UBy2ht/7h2+e+t9uBUHJmdmye5HDFjTP3f+LSqb9cvylcgl3UvNhOaThssHr1U8PV
q58aXvadn7UmbjqdGov3ml8ffeBe9VGH7ds59NAlnQOW7tnZZ/GCasGi+dVue8yr5u4+h8Z3G6fu
nA66nYoqIlRpViRFoqf1Dhy3j02rRznNOEW/KjOHgDBoMJgehN7kTJjZMsVTGyfD1g1bw5bHN4Zn
Hn5y+Pj964erHlg/vGPtpubu3pBfCmD+Ek47aFFYRVhw6qu77zj9+LEl3rMeAvDEU6F/wRXT13/l
ypmP94d8y46ozwswDYcN1q7f3Kxdv7m55qcPzZrws2lHph0UR3H0QdXbzv393X6LqijTmiHzXauH
m879+vQlV986+DRjNkhmNs2mXTltd6DodrDwt35zztsOXtLZe3o6hB/8tL/qoxdNfe6htS8582I2
zaYXbNruQHHYfvVvnnh458BP/O3kP/7Vd2b+63SP79jeZc6m2TSbnt/0/wGLUkg80O9A4QAAAABJ
RU5ErkJggg==
"
id="image3810"
x="9.712183"
y="3.7505856" /></svg>
Side by Side

After

Width:  |  Height:  |  Size: 50 KiB

1
src/khoj/interface/web/assets/icons/confirm-icon.svg Normal file
View File

@@ -0,0 +1 @@
<svg id="Layer_1" data-name="Layer 1" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 122.88 122.88"><defs><style>.cls-1{fill:#00a912;}.cls-1,.cls-2{fill-rule:evenodd;}.cls-2{fill:#fff;}</style></defs><title>confirm</title><path class="cls-1" d="M61.44,0A61.44,61.44,0,1,1,0,61.44,61.44,61.44,0,0,1,61.44,0Z"/><path class="cls-2" d="M42.37,51.68,53.26,62,79,35.87c2.13-2.16,3.47-3.9,6.1-1.19l8.53,8.74c2.8,2.77,2.66,4.4,0,7L58.14,85.34c-5.58,5.46-4.61,5.79-10.26.19L28,65.77c-1.18-1.28-1.05-2.57.24-3.84l9.9-10.27c1.5-1.58,2.7-1.44,4.22,0Z"/></svg>
Side by Side

After

Width:  |  Height:  |  Size: 549 B

BIN
src/khoj/interface/web/assets/icons/favicon-128x128.ico Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 200 KiB

BIN
src/khoj/interface/web/assets/icons/favicon-128x128.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 12 KiB

BIN
src/khoj/interface/web/assets/icons/favicon.icns Normal file
View File

Binary file not shown.

1
src/khoj/interface/web/assets/icons/github.svg Normal file
View File

@@ -0,0 +1 @@
<svg width="98" height="96" xmlns="http://www.w3.org/2000/svg"><path fill-rule="evenodd" clip-rule="evenodd" d="M48.854 0C21.839 0 0 22 0 49.217c0 21.756 13.993 40.172 33.405 46.69 2.427.49 3.316-1.059 3.316-2.362 0-1.141-.08-5.052-.08-9.127-13.59 2.934-16.42-5.867-16.42-5.867-2.184-5.704-5.42-7.17-5.42-7.17-4.448-3.015.324-3.015.324-3.015 4.934.326 7.523 5.052 7.523 5.052 4.367 7.496 11.404 5.378 14.235 4.074.404-3.178 1.699-5.378 3.074-6.6-10.839-1.141-22.243-5.378-22.243-24.283 0-5.378 1.94-9.778 5.014-13.2-.485-1.222-2.184-6.275.486-13.038 0 0 4.125-1.304 13.426 5.052a46.97 46.97 0 0 1 12.214-1.63c4.125 0 8.33.571 12.213 1.63 9.302-6.356 13.427-5.052 13.427-5.052 2.67 6.763.97 11.816.485 13.038 3.155 3.422 5.015 7.822 5.015 13.2 0 18.905-11.404 23.06-22.324 24.283 1.78 1.548 3.316 4.481 3.316 9.126 0 6.6-.08 11.897-.08 13.526 0 1.304.89 2.853 3.316 2.364 19.412-6.52 33.405-24.935 33.405-46.691C97.707 22 75.788 0 48.854 0z" fill="#24292f"/></svg>
Side by Side

After

Width:  |  Height:  |  Size: 964 B

BIN
src/khoj/interface/web/assets/icons/khoj-logo-sideways-200.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 13 KiB

5385
src/khoj/interface/web/assets/icons/khoj-logo-sideways.svg Normal file
View File

File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 1.2 MiB

1
src/khoj/interface/web/assets/icons/logotype.svg Normal file
View File

@@ -0,0 +1 @@
<svg id="Layer_1" data-name="Layer 1" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 942.19 656.92"><defs><style>.cls-1{fill:#1e1d1e;}</style></defs><title>logotype</title><path class="cls-1" d="M122.48,234.63c7.25,7.25,3.29,38.21,3.29,57.32,0,11.86,2,21.74,2,31,0,9.88-2,19.76-2,29,2,7.9,0,36.89,4,38.87,5.93,4,11.86-15.81,13.83-17.79,5.93-13.18,2-5.27,9.23-21.08,7.9-15.16,15.81-31,21.74-42.17,5.27-11.86,13.18-29,19.11-42.83,5.93-9.22,9.88-27,19.11-34.26,5.92-5.93,25-5.93,42.82-5.93,13.18,2,27,2,31,9.22,5.27,11.86-15.82,42.83-21.74,56-7.25,15.81-13.18,25.7-21.09,38.87-2,4-2,7.91-4,13.84-7.25,19.11-17.13,32.28-27,52-4,7.25-7.25,17.13-13.18,25-4,11.2-9.89,21.08-9.89,29,0,11.2,7.91,27,11.86,34.26,13.18,25.7,27,50.73,40.19,77.75,2,5.27,2,11.2,5.93,17.13,5.93,15.15,13.84,27,23.06,46.12,5.94,13.84,21.09,38.87,17.79,50.07-5.93,13.84-21.74,9.88-38.87,9.88-21.08,0-36.89,0-44.14-4-7.91-5.93-13.84-27-19.77-38.21-13.18-31-19.11-48.76-34.26-81a152.72,152.72,0,0,0-15.81-33c-7.91-9.88-7.91,21.09-9.89,34.92v38.22c0,25,5.93,67.86-5.27,79.06-11.86,9.89-67.86,7.91-75.77-5.93-2-7.24,0-30.31,0-50.07,2-19.11,0-31,0-48.1-2-42.17-2-81-2-123.21,0-15.81,4-31,2-42.83,0-7.9-2-13.17-2-21.08,0-5.93,2-9.88,2-15.81,0-13.18-2-27-2-40.19,0-13.84,2-27,2-38.88,0-17.13-5.27-38.21-2-52,2-5.93,7.91-11.2,9.89-13.18,2,0,2,0,4-2C71.74,222.77,114.57,224.74,122.48,234.63Z" transform="translate(-41.67 -218.04)"/><path class="cls-1" d="M547.58,232.65c3.95,2,3.95,9.23,5.93,17.13,2,15.16,0,46.12,0,63.91-2,29,0,59.3,0,81V523.87c0,34.92,0,83-2,123.87,0,17.13,2,30.31-5.93,36.24-9.22,5.93-63.91,5.93-71.16,0-7.9-7.91-3.95-33-3.95-50.08V610.84c0-36.89-2-71.81,0-114,0-9.88,2-21.08,0-25-5.93-5.93-21.08-2-32.94-2-7.91,0-25-5.93-32.95,2-7.25,9.88-4,36.9-4,54,0,42.17,2,63.91,2,108,0,23.07,2,42.17-9.23,50.08-11.86,7.91-58,5.93-65.89-2-11.86-11.2-9.88-56-9.88-83,0-83,3.95-144.29,2-218.08,0-15.16,2-31,2-42.17,0-7.91-2-15.81-3.95-27,0-5.93,2-13.84,2-19.77,2-21.08-2-48.1,2-56,7.91-11.2,67.87-13.17,77.09-1.31,7.91,11.2,2,34.26,2,48.09-2,15.16,0,34.93,0,56v27c0,9.23,0,23.06,2,27,3.95,5.93,23.71,5.93,32.94,5.93,7.9,0,29,2,32.94-4s4-38.87,4-50.07V264.94c0-11.2-2-23.06,2-30.31,3.95-7.91,13.18-5.93,21.08-7.91C501.46,226.72,539.67,222.77,547.58,232.65Z" transform="translate(-41.67 -218.04)"/><path class="cls-1" d="M716,218.81c31-2,61.93,7.91,81,17.13,5.93,4,19.11,17.8,23.06,23.72,9.88,13.18,15.81,23.06,19.11,40.2,2,3.95,3.95,5.93,3.95,7.9v7.91c2,9.22,5.93,19.11,7.91,25v21.08c0,5.93,2,11.2,2,17.13,4,34.92,2,75.77,0,114,0,25,0,50.07-3.95,73.13-4,23.72-9.88,48.76-15.15,61.94-2,3.95-7.91,13.83-11.86,17.78l-2,2c-5.93,9.23-17.13,17.13-29,25-5.93,3.29-7.91,5.27-13.18,7.25-5.93,2-11.86,2-21.74,4-5.27,0-11.2,3.95-17.13,3.95-13.18,2-31,2-40.19,2-11.86-2-25.7-5.93-36.9-11.86-2,0-4-3.29-4-3.29a107,107,0,0,1-17.14-7.91c-7.9-5.93-17.78-17.79-23.71-25-13.18-23.06-23.07-58-25-90.92-2-25-2-58-2-79.07-2-15.81,0-32.94,2-48.1,0-21.74,0-44.8,2-63.91,2-9.88,4-19.1,5.94-27,0-11.86,0-21.08,2-31,5.93-17.13,19.11-46.12,33-58,3.95-2,9.22-5.93,15.15-9.22,15.81-7.91,38.87-13.84,63.91-15.82ZM677.11,348c-5.93,48.76-4,100.81-2,150.88,2,38.88,4,98.18,21.74,112,0,2,5.27,4,9.22,5.93,15.82,5.28,29,2,40.86-5.93,3.95-4,9.22-9.88,11.19-13.83,5.93-9.23,9.89-36.24,11.86-54,0-15.15,0-34.26,2-56,0-42.17-2-85-5.93-121.24-2-25-3.95-48.09-11.86-59.95C748.93,297.88,741,292,733.12,290c-9.23-3.29-13.18,0-17.13,0h-2C689,293.93,681.07,319,677.11,348Z" transform="translate(-41.67 -218.04)"/><path class="cls-1" d="M975.89,226.44c9.23,9.23,7.91,234.41,7.91,263.4-2,63.25-2,108.05-2,175.26,0,11.86,2,17.79,2,25.69,0,15.16-3.95,38.22-2,61.28,0,23.06,2,51.24-4,64.42-4,17.13-19.77,36.89-34.92,46.12-15.15,9.88-40.85,13.84-71.16,11.86-19.76-2-42.82-4-50.73-15.82-2-5.27-2-21.08-2-32.28,0-15.81,0-27,5.93-31,11.86-5.93,36.9,13.18,56,7.91,7.9-2,15.81-9.88,19.76-23.72,1.32-11.2,0-27.52,0-40.7,0-11.86,1.32-21.08,1.32-32.94V634.79c0-29-3.29-56-3.29-81.7,0-36.24,2-253.51,2-286.46,0-23.06-7.91-38.21,9.22-46.12C923.84,216.56,966,216.56,975.89,226.44Z" transform="translate(-41.67 -218.04)"/></svg>
Side by Side

After

Width:  |  Height:  |  Size: 3.9 KiB

1
src/khoj/interface/web/assets/icons/markdown.svg Normal file
View File

@@ -0,0 +1 @@
<svg xmlns="http://www.w3.org/2000/svg" width="208" height="128" viewBox="0 0 208 128"><rect width="198" height="118" x="5" y="5" ry="10" stroke="#000" stroke-width="10" fill="none"/><path d="M30 98V30h20l20 25 20-25h20v68H90V59L70 84 50 59v39zm125 0l-30-33h20V30h20v35h20z"/></svg>
Side by Side

After

Width:  |  Height:  |  Size: 283 B

4
src/khoj/interface/web/assets/icons/notion.svg Normal file
View File

@@ -0,0 +1,4 @@
<svg width="100" height="100" viewBox="0 0 100 100" fill="none" xmlns="http://www.w3.org/2000/svg">
<path d="M6.017 4.313l55.333 -4.087c6.797 -0.583 8.543 -0.19 12.817 2.917l17.663 12.443c2.913 2.14 3.883 2.723 3.883 5.053v68.243c0 4.277 -1.553 6.807 -6.99 7.193L24.467 99.967c-4.08 0.193 -6.023 -0.39 -8.16 -3.113L3.3 79.94c-2.333 -3.113 -3.3 -5.443 -3.3 -8.167V11.113c0 -3.497 1.553 -6.413 6.017 -6.8z" fill="#fff"/>
<path fill-rule="evenodd" clip-rule="evenodd" d="M61.35 0.227l-55.333 4.087C1.553 4.7 0 7.617 0 11.113v60.66c0 2.723 0.967 5.053 3.3 8.167l13.007 16.913c2.137 2.723 4.08 3.307 8.16 3.113l64.257 -3.89c5.433 -0.387 6.99 -2.917 6.99 -7.193V20.64c0 -2.21 -0.873 -2.847 -3.443 -4.733L74.167 3.143c-4.273 -3.107 -6.02 -3.5 -12.817 -2.917zM25.92 19.523c-5.247 0.353 -6.437 0.433 -9.417 -1.99L8.927 11.507c-0.77 -0.78 -0.383 -1.753 1.557 -1.947l53.193 -3.887c4.467 -0.39 6.793 1.167 8.54 2.527l9.123 6.61c0.39 0.197 1.36 1.36 0.193 1.36l-54.933 3.307 -0.68 0.047zM19.803 88.3V30.367c0 -2.53 0.777 -3.697 3.103 -3.893L86 22.78c2.14 -0.193 3.107 1.167 3.107 3.693v57.547c0 2.53 -0.39 4.67 -3.883 4.863l-60.377 3.5c-3.493 0.193 -5.043 -0.97 -5.043 -4.083zm59.6 -54.827c0.387 1.75 0 3.5 -1.75 3.7l-2.91 0.577v42.773c-2.527 1.36 -4.853 2.137 -6.797 2.137 -3.107 0 -3.883 -0.973 -6.21 -3.887l-19.03 -29.94v28.967l6.02 1.363s0 3.5 -4.857 3.5l-13.39 0.777c-0.39 -0.78 0 -2.723 1.357 -3.11l3.497 -0.97v-38.3L30.48 40.667c-0.39 -1.75 0.58 -4.277 3.3 -4.473l14.367 -0.967 19.8 30.327v-26.83l-5.047 -0.58c-0.39 -2.143 1.163 -3.7 3.103 -3.89l13.4 -0.78z" fill="#000"/>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 1.5 KiB

1
src/khoj/interface/web/assets/icons/org.svg Normal file
View File

File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 7.8 KiB

23
src/khoj/interface/web/assets/icons/pdf.svg Normal file
View File

@@ -0,0 +1,23 @@
<?xml version="1.0" encoding="UTF-8"?>
<svg enable-background="new 0 0 334.371 380.563" version="1.1" viewBox="0 0 14 16" xml:space="preserve" xmlns="http://www.w3.org/2000/svg">
<g transform="matrix(.04589 0 0 .04589 -.66877 -.73379)">
<polygon points="51.791 356.65 51.791 23.99 204.5 23.99 282.65 102.07 282.65 356.65" fill="#fff" stroke-width="212.65"/>
<path d="m201.19 31.99 73.46 73.393v243.26h-214.86v-316.66h141.4m6.623-16h-164.02v348.66h246.85v-265.9z" stroke-width="21.791"/>
</g>
<g transform="matrix(.04589 0 0 .04589 -.66877 -.73379)">
<polygon points="282.65 356.65 51.791 356.65 51.791 23.99 204.5 23.99 206.31 25.8 206.31 100.33 280.9 100.33 282.65 102.07" fill="#fff" stroke-width="212.65"/>
<path d="m198.31 31.99v76.337h76.337v240.32h-214.86v-316.66h138.52m9.5-16h-164.02v348.66h246.85v-265.9l-6.43-6.424h-69.907v-69.842z" stroke-width="21.791"/>
</g>
<g transform="matrix(.04589 0 0 .04589 -.66877 -.73379)" stroke-width="21.791">
<polygon points="258.31 87.75 219.64 87.75 219.64 48.667 258.31 86.38"/>
<path d="m227.64 67.646 12.41 12.104h-12.41v-12.104m-5.002-27.229h-10.998v55.333h54.666v-12.742z"/>
</g>
<g transform="matrix(.04589 0 0 .04589 -.66877 -.73379)" fill="#ed1c24" stroke-width="212.65">
<polygon points="311.89 284.49 22.544 284.49 22.544 167.68 37.291 152.94 37.291 171.49 297.15 171.49 297.15 152.94 311.89 167.68"/>
<path d="m303.65 168.63 1.747 1.747v107.62h-276.35v-107.62l1.747-1.747v9.362h272.85v-9.362m-12.999-31.385v27.747h-246.86v-27.747l-27.747 27.747v126h302.35v-126z"/>
</g>
<rect x="1.7219" y="7.9544" width="10.684" height="4.0307" fill="none"/>
<g transform="matrix(.04589 0 0 .04589 1.7219 11.733)" fill="#fff" stroke-width="21.791"><path d="m9.216 0v-83.2h30.464q6.784 0 12.928 1.408 6.144 1.28 10.752 4.608 4.608 3.2 7.296 8.576 2.816 5.248 2.816 13.056 0 7.68-2.816 13.184-2.688 5.504-7.296 9.088-4.608 3.456-10.624 5.248-6.016 1.664-12.544 1.664h-8.96v26.368zm22.016-43.776h7.936q6.528 0 9.6-3.072 3.2-3.072 3.2-8.704t-3.456-7.936-9.856-2.304h-7.424z"/><path d="m87.04 0v-83.2h24.576q9.472 0 17.28 2.304 7.936 2.304 13.568 7.296t8.704 12.8q3.2 7.808 3.2 18.816t-3.072 18.944-8.704 13.056q-5.504 5.12-13.184 7.552-7.552 2.432-16.512 2.432zm22.016-17.664h1.28q4.48 0 8.448-1.024 3.968-1.152 6.784-3.84 2.944-2.688 4.608-7.424t1.664-12.032-1.664-11.904-4.608-7.168q-2.816-2.56-6.784-3.456-3.968-1.024-8.448-1.024h-1.28z"/><path d="m169.22 0v-83.2h54.272v18.432h-32.256v15.872h27.648v18.432h-27.648v30.464z"/></g>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 2.4 KiB

101
src/khoj/interface/web/assets/khoj.css Normal file
View File

@@ -0,0 +1,101 @@
/* Amber Light scheme (Default) */
/* Can be forced with data-theme="light" */
[data-theme="light"],
:root:not([data-theme="dark"]) {
--primary: #ffb300;
--primary-hover: #ffa000;
--primary-focus: rgba(255, 179, 0, 0.125);
--primary-inverse: rgba(0, 0, 0, 0.75);
}
/* Amber Dark scheme (Auto) */
/* Automatically enabled if user has Dark mode enabled */
@media only screen and (prefers-color-scheme: dark) {
:root:not([data-theme]) {
--primary: #ffb300;
--primary-hover: #ffc107;
--primary-focus: rgba(255, 179, 0, 0.25);
--primary-inverse: rgba(0, 0, 0, 0.75);
}
}
/* Amber Dark scheme (Forced) */
/* Enabled if forced with data-theme="dark" */
[data-theme="dark"] {
--primary: #ffb300;
--primary-hover: #ffc107;
--primary-focus: rgba(255, 179, 0, 0.25);
--primary-inverse: rgba(0, 0, 0, 0.75);
}
/* Amber (Common styles) */
:root {
--form-element-active-border-color: var(--primary);
--form-element-focus-color: var(--primary-focus);
--switch-color: var(--primary-inverse);
--switch-checked-background-color: var(--primary);
}
.khoj-configure {
display: grid;
grid-template-columns: 1fr;
padding: 0 24px;
}
.khoj-header {
display: grid;
grid-auto-flow: column;
gap: 20px;
padding: 16px 0;
margin: 0 0 16px 0;
}
nav.khoj-nav {
display: grid;
grid-auto-flow: column;
grid-gap: 32px;
justify-self: right;
}
a.khoj-nav {
display: flex;
align-items: center;
}
a.khoj-logo {
justify-self: left;
}
.khoj-nav a {
color: #333;
text-decoration: none;
font-size: 20px;
font-weight: normal;
padding: 0 4px;
border-radius: 4px;
justify-self: center;
margin: 0;
}
.khoj-nav a:hover {
background-color: var(--primary-hover);
}
.khoj-nav-selected {
background-color: var(--primary);
}
img.khoj-logo {
width: min(60vw, 111px);
max-width: 100%;
justify-self: center;
}
@media only screen and (max-width: 600px) {
div.khoj-header {
display: grid;
grid-auto-flow: column;
gap: 20px;
padding: 16px 10px;
margin: 0 0 16px 0;
}
nav.khoj-nav {
grid-gap: 0px;
justify-content: space-between;
}
}

236
src/interface/web/assets/markdown-it.min.js → src/khoj/interface/web/assets/markdown-it.min.js vendored
View File

@@ -1,6 +1,6 @@
/*! markdown-it 13.0.1 https://github.com/markdown-it/markdown-it @license MIT */
(function(global, factory) {
typeof exports === "object" && typeof module !== "undefined" ? module.exports = factory() : typeof define === "function" && define.amd ? define(factory) : (global = typeof globalThis !== "undefined" ? globalThis : global || self,
typeof exports === "object" && typeof module !== "undefined" ? module.exports = factory() : typeof define === "function" && define.amd ? define(factory) : (global = typeof globalThis !== "undefined" ? globalThis : global || self,
global.markdownit = factory());
})(this, (function() {
"use strict";
@@ -2164,7 +2164,7 @@
var encodeCache = {};
// Create a lookup array where anything but characters in `chars` string
// and alphanumeric chars is percent-encoded.
function getEncodeCache(exclude) {
var i, ch, cache = encodeCache[exclude];
if (cache) {
@@ -2187,11 +2187,11 @@
}
// Encode unsafe characters with percent-encoding, skipping already
// encoded sequences.
// - string - string to encode
// - exclude - list of characters to ignore (in addition to a-zA-Z0-9)
// - keepEscaped - don't encode '%' in a correct escape sequence (default: true)
function encode$2(string, exclude, keepEscaped) {
var i, l, code, nextCode, cache, result = "";
if (typeof exclude !== "string") {
@@ -2253,7 +2253,7 @@
return cache;
}
// Decode percent-encoded string.
function decode$2(string, exclude) {
var cache;
if (typeof exclude !== "string") {
@@ -2340,26 +2340,26 @@
return result;
};
// Copyright Joyent, Inc. and other Node contributors.
// Changes from joyent/node:
// 1. No leading slash in paths,
// e.g. in `url.parse('http://foo?bar')` pathname is ``, not `/`
// 2. Backslashes are not replaced with slashes,
// so `http:\\example.org\` is treated like a relative path
// 3. Trailing colon is treated like a part of the path,
// i.e. in `http://example.org:foo` pathname is `:foo`
// 4. Nothing is URL-encoded in the resulting object,
// (in joyent/node some chars in auth and paths are encoded)
// 5. `url.parse()` does not have `parseQueryString` argument
// 6. Removed extraneous result properties: `host`, `path`, `query`, etc.,
// which can be constructed using other parts of the url.
function Url() {
this.protocol = null;
this.slashes = null;
@@ -2373,28 +2373,28 @@
// Reference: RFC 3986, RFC 1808, RFC 2396
// define these here so at least they only have to be
// compiled once on the first module load.
var protocolPattern = /^([a-z0-9.+-]+:)/i, portPattern = /:[0-9]*$/,
var protocolPattern = /^([a-z0-9.+-]+:)/i, portPattern = /:[0-9]*$/,
// Special case for a simple path URL
simplePathPattern = /^(\/\/?(?!\/)[^\?\s]*)(\?[^\s]*)?$/,
simplePathPattern = /^(\/\/?(?!\/)[^\?\s]*)(\?[^\s]*)?$/,
// RFC 2396: characters reserved for delimiting URLs.
// We actually just auto-escape these.
delims = [ "<", ">", '"', "`", " ", "\r", "\n", "\t" ],
delims = [ "<", ">", '"', "`", " ", "\r", "\n", "\t" ],
// RFC 2396: characters not allowed for various reasons.
unwise = [ "{", "}", "|", "\\", "^", "`" ].concat(delims),
unwise = [ "{", "}", "|", "\\", "^", "`" ].concat(delims),
// Allowed by RFCs, but cause of XSS attacks. Always escape these.
autoEscape = [ "'" ].concat(unwise),
autoEscape = [ "'" ].concat(unwise),
// Characters that are never ever allowed in a hostname.
// Note that any invalid chars are also handled, but these
// are the ones that are *expected* to be seen, so we fast-path
// them.
nonHostChars = [ "%", "/", "?", ";", "#" ].concat(autoEscape), hostEndingChars = [ "/", "?", "#" ], hostnameMaxLen = 255, hostnamePartPattern = /^[+a-z0-9A-Z_-]{0,63}$/, hostnamePartStart = /^([+a-z0-9A-Z_-]{0,63})(.*)$/,
nonHostChars = [ "%", "/", "?", ";", "#" ].concat(autoEscape), hostEndingChars = [ "/", "?", "#" ], hostnameMaxLen = 255, hostnamePartPattern = /^[+a-z0-9A-Z_-]{0,63}$/, hostnamePartStart = /^([+a-z0-9A-Z_-]{0,63})(.*)$/,
// protocols that can allow "unsafe" and "unwise" chars.
/* eslint-disable no-script-url */
// protocols that never have a hostname.
hostlessProtocol = {
javascript: true,
"javascript:": true
},
},
// protocols that always contain a // bit.
slashedProtocol = {
http: true,
@@ -2632,7 +2632,7 @@
return _hasOwnProperty.call(object, key);
}
// Merge objects
function assign(obj /*from1, from2, from3, ...*/) {
var sources = Array.prototype.slice.call(arguments, 1);
sources.forEach((function(source) {
@@ -2798,12 +2798,12 @@
return regex$4.test(ch);
}
// Markdown ASCII punctuation characters.
// !, ", #, $, %, &, ', (, ), *, +, ,, -, ., /, :, ;, <, =, >, ?, @, [, \, ], ^, _, `, {, |, }, or ~
// http://spec.commonmark.org/0.15/#ascii-punctuation-character
// Don't confuse with unicode punctuation !!! It lacks some chars in ascii range.
function isMdAsciiPunct(ch) {
switch (ch) {
case 33 /* ! */ :
@@ -2845,58 +2845,58 @@
}
}
// Hepler to unify [reference labels].
function normalizeReference(str) {
// Trim and collapse whitespace
str = str.trim().replace(/\s+/g, " ");
// In node v10 'ẞ'.toLowerCase() === 'Ṿ', which is presumed to be a bug
// fixed in v12 (couldn't find any details).
// So treat this one as a special case
// (remove this when node v10 is no longer supported).
if ("\u1e9e".toLowerCase() === "\u1e7e") {
str = str.replace(/\u1e9e/g, "\xdf");
}
// .toLowerCase().toUpperCase() should get rid of all differences
// between letter variants.
// Simple .toLowerCase() doesn't normalize 125 code points correctly,
// and .toUpperCase doesn't normalize 6 of them (list of exceptions:
// İ, ϴ, ẞ, Ω, K, Å - those are already uppercased, but have differently
// uppercased versions).
// Here's an example showing how it happens. Lets take greek letter omega:
// uppercase U+0398 (Θ), U+03f4 (ϴ) and lowercase U+03b8 (θ), U+03d1 (ϑ)
// Unicode entries:
// 0398;GREEK CAPITAL LETTER THETA;Lu;0;L;;;;;N;;;;03B8;
// 03B8;GREEK SMALL LETTER THETA;Ll;0;L;;;;;N;;;0398;;0398
// 03D1;GREEK THETA SYMBOL;Ll;0;L;<compat> 03B8;;;;N;GREEK SMALL LETTER SCRIPT THETA;;0398;;0398
// 03F4;GREEK CAPITAL THETA SYMBOL;Lu;0;L;<compat> 0398;;;;N;;;;03B8;
// Case-insensitive comparison should treat all of them as equivalent.
// But .toLowerCase() doesn't change ϑ (it's already lowercase),
// and .toUpperCase() doesn't change ϴ (already uppercase).
// Applying first lower then upper case normalizes any character:
// '\u0398\u03f4\u03b8\u03d1'.toLowerCase().toUpperCase() === '\u0398\u0398\u0398\u0398'
// Note: this is equivalent to unicode case folding; unicode normalization
// is a different step that is not required here.
// Final result should be uppercased, because it's later stored in an object
// (this avoid a conflict with Object.prototype members,
// most notably, `__proto__`)
return str.toLowerCase().toUpperCase();
}
////////////////////////////////////////////////////////////////////////////////
// Re-export libraries commonly used in both markdown-it and its plugins,
// so plugins won't have to depend on them explicitly, which reduces their
// bundled size (e.g. a browser build).
exports.lib = {};
exports.lib.mdurl = mdurl;
exports.lib.ucmicro = uc_micro;
@@ -3129,7 +3129,7 @@
var token = tokens[idx];
// "alt" attr MUST be set, even if empty. Because it's mandatory and
// should be placed on proper position for tests.
// Replace content with actual value
token.attrs[token.attrIndex("alt")][1] = slf.renderInlineAsText(token.children, options, env);
return slf.renderToken(tokens, idx, options);
@@ -3215,11 +3215,11 @@
}
// Insert a newline between hidden paragraph and subsequent opening
// block-level tag.
// For example, here we should insert a newline before blockquote:
// - a
// >
if (token.block && token.nesting !== -1 && idx && tokens[idx - 1].hidden) {
result += "\n";
}
@@ -3343,16 +3343,16 @@
// }
this.__rules__ = [];
// Cached rule chains.
// First level - chain name, '' for default.
// Second level - diginal anchor for fast filtering by charcodes.
this.__cache__ = null;
}
////////////////////////////////////////////////////////////////////////////////
// Helper methods, should not be used directly
// Find rule index by name
Ruler.prototype.__find__ = function(name) {
for (var i = 0; i < this.__rules__.length; i++) {
if (this.__rules__[i].name === name) {
@@ -3362,7 +3362,7 @@
return -1;
};
// Build rules lookup cache
Ruler.prototype.__compile__ = function() {
var self = this;
var chains = [ "" ];
@@ -3726,7 +3726,7 @@
// Linkifier might send raw hostnames like "example.com", where url
// starts with domain name. So we prepend http:// in those cases,
// and remove it afterwards.
if (!links[ln].schema) {
urlText = state.md.normalizeLinkText("http://" + urlText).replace(/^http:\/\//, "");
} else if (links[ln].schema === "mailto:" && !/^mailto:/i.test(urlText)) {
@@ -3874,7 +3874,7 @@
isSingle = t[0] === "'";
// Find previous character,
// default to space if it's the beginning of the line
lastChar = 32;
if (t.index - 1 >= 0) {
lastChar = text.charCodeAt(t.index - 1);
@@ -3890,7 +3890,7 @@
}
// Find next character,
// default to space if it's the end of the line
nextChar = 32;
if (pos < max) {
nextChar = text.charCodeAt(pos);
@@ -4193,7 +4193,7 @@
// re-export Token class to use in core rules
StateCore.prototype.Token = token;
var state_core = StateCore;
var _rules$2 = [ [ "normalize", normalize ], [ "block", block ], [ "inline", inline ], [ "linkify", linkify$1 ], [ "replacements", replacements ], [ "smartquotes", smartquotes ],
var _rules$2 = [ [ "normalize", normalize ], [ "block", block ], [ "inline", inline ], [ "linkify", linkify$1 ], [ "replacements", replacements ], [ "smartquotes", smartquotes ],
// `text_join` finds `text_special` tokens (for escape sequences)
// and joins them with the rest of the text
[ "text_join", text_join ] ];
@@ -4590,12 +4590,12 @@
oldParentType = state.parentType;
state.parentType = "blockquote";
// Search the end of the block
// Block ends with either:
// 1. an empty line outside:
// ```
// > test
// ```
// 2. an empty line inside:
// ```
@@ -4712,7 +4712,7 @@
oldTShift.push(state.tShift[nextLine]);
oldSCount.push(state.sCount[nextLine]);
// A negative indentation means that this is a paragraph continuation
state.sCount[nextLine] = -1;
}
oldIndent = state.blkIndent;
@@ -4905,9 +4905,9 @@
}
token.map = listLines = [ startLine, 0 ];
token.markup = String.fromCharCode(markerCharCode);
// Iterate list items
nextLine = startLine;
prevEmptyEnd = false;
terminatorRules = state.md.block.ruler.getRules("list");
@@ -4957,7 +4957,7 @@
// - example list
// ^ listIndent position will be here
// ^ blkIndent position will be here
oldListIndent = state.listIndent;
state.listIndent = state.blkIndent;
state.blkIndent = indent;
@@ -4995,9 +4995,9 @@
if (nextLine >= endLine) {
break;
}
// Try to check if list is terminated or continued.
if (state.sCount[nextLine] < state.blkIndent) {
break;
}
@@ -5245,7 +5245,7 @@
var HTML_OPEN_CLOSE_TAG_RE = html_re.HTML_OPEN_CLOSE_TAG_RE;
// An array of opening and corresponding closing sequences for html tags,
// last argument defines whether it can terminate a paragraph or not
var HTML_SEQUENCES = [ [ /^<(script|pre|style|textarea)(?=(\s|>|$))/i, /<\/(script|pre|style|textarea)>/i, true ], [ /^<!--/, /-->/, true ], [ /^<\?/, /\?>/, true ], [ /^<![A-Z]/, />/, true ], [ /^<!\[CDATA\[/, /\]\]>/, true ], [ new RegExp("^</?(" + html_blocks.join("|") + ")(?=(\\s|/?>|$))", "i"), /^$/, true ], [ new RegExp(HTML_OPEN_CLOSE_TAG_RE.source + "\\s*$"), /^$/, false ] ];
var html_block = function html_block(state, startLine, endLine, silent) {
var i, nextLine, token, lineText, pos = state.bMarks[startLine] + state.tShift[startLine], max = state.eMarks[startLine];
@@ -5357,9 +5357,9 @@
if (state.sCount[nextLine] - state.blkIndent > 3) {
continue;
}
// Check for underline in setext header
if (state.sCount[nextLine] >= state.blkIndent) {
pos = state.bMarks[nextLine] + state.tShift[nextLine];
max = state.eMarks[nextLine];
@@ -5456,9 +5456,9 @@
// link to parser instance
this.md = md;
this.env = env;
// Internal state vartiables
this.tokens = tokens;
this.bMarks = [];
// line begin offsets for fast jumps
@@ -5470,14 +5470,14 @@
// indents for each line (tabs expanded)
// An amount of virtual spaces (tabs expanded) between beginning
// of each line (bMarks) and real beginning of that line.
// It exists only as a hack because blockquotes override bMarks
// losing information in the process.
// It's used only when expanding tabs, you can think about it as
// an initial tab length, e.g. bsCount=21 applied to string `\t123`
// means first tab should be expanded to 4-21%4 === 3 spaces.
this.bsCount = [];
// block parser variables
this.blkIndent = 0;
@@ -5543,7 +5543,7 @@
// don't count last fake line
}
// Push new token to "stream".
StateBlock.prototype.push = function(type, tag, nesting) {
var token$1 = new token(type, tag, nesting);
token$1.block = true;
@@ -5655,7 +5655,7 @@
// re-export Token class to use in block rules
StateBlock.prototype.Token = token;
var state_block = StateBlock;
var _rules$1 = [
var _rules$1 = [
// First 2 params - rule name & source. Secondary array - list of rules,
// which can be terminated by this one.
[ "table", table, [ "paragraph", "reference" ] ], [ "code", code ], [ "fence", fence, [ "paragraph", "reference", "blockquote", "list" ] ], [ "blockquote", blockquote, [ "paragraph", "reference", "blockquote", "list" ] ], [ "hr", hr, [ "paragraph", "reference", "blockquote", "list" ] ], [ "list", list, [ "paragraph", "reference", "blockquote" ] ], [ "reference", reference ], [ "html_block", html_block, [ "paragraph", "reference", "blockquote" ] ], [ "heading", heading, [ "paragraph", "reference", "blockquote" ] ], [ "lheading", lheading ], [ "paragraph", paragraph ] ];
@@ -5675,7 +5675,7 @@
}
}
// Generate tokens for input range
ParserBlock.prototype.tokenize = function(state, startLine, endLine) {
var ok, i, rules = this.ruler.getRules(""), len = rules.length, line = startLine, hasEmptyLines = false, maxNesting = state.md.options.maxNesting;
while (line < endLine) {
@@ -5696,7 +5696,7 @@
}
// Try all possible rules.
// On success, rule should:
// - update `state.line`
// - update `state.tokens`
// - return true
@@ -5961,7 +5961,7 @@
};
// ~~strike through~~
// Insert each marker as a separate text token, and add it to delimiter list
var tokenize$1 = function strikethrough(state, silent) {
var i, scanned, token, len, ch, start = state.pos, marker = state.src.charCodeAt(start);
if (silent) {
@@ -6027,9 +6027,9 @@
// If a marker sequence has an odd number of characters, it's splitted
// like this: `~~~~~` -> `~` + `~~` + `~~`, leaving one marker at the
// start of the sequence.
// So, we have to move all those markers after subsequent s_close tags.
while (loneMarkers.length) {
i = loneMarkers.pop();
j = i + 1;
@@ -6045,7 +6045,7 @@
}
}
// Walk through delimiter list and replace text tokens with tags
var postProcess_1$1 = function strikethrough(state) {
var curr, tokens_meta = state.tokens_meta, max = state.tokens_meta.length;
postProcess$1(state, state.delimiters);
@@ -6061,7 +6061,7 @@
};
// Process *this* and _that_
// Insert each marker as a separate text token, and add it to delimiter list
var tokenize = function emphasis(state, silent) {
var i, scanned, token, start = state.pos, marker = state.src.charCodeAt(start);
if (silent) {
@@ -6107,12 +6107,12 @@
endDelim = delimiters[startDelim.end];
// If the previous delimiter has the same marker and is adjacent to this one,
// merge those into one strong delimiter.
// `<em><em>whatever</em></em>` -> `<strong>whatever</strong>`
isStrong = i > 0 && delimiters[i - 1].end === startDelim.end + 1 &&
isStrong = i > 0 && delimiters[i - 1].end === startDelim.end + 1 &&
// check that first two markers match and adjacent
delimiters[i - 1].marker === startDelim.marker && delimiters[i - 1].token === startDelim.token - 1 &&
delimiters[i - 1].marker === startDelim.marker && delimiters[i - 1].token === startDelim.token - 1 &&
// check that last two markers are adjacent (we can safely assume they match)
delimiters[startDelim.end + 1].token === endDelim.token + 1;
ch = String.fromCharCode(startDelim.marker);
@@ -6136,7 +6136,7 @@
}
}
// Walk through delimiter list and replace text tokens with tags
var postProcess_1 = function emphasis(state) {
var curr, tokens_meta = state.tokens_meta, max = state.tokens_meta.length;
postProcess(state, state.delimiters);
@@ -6251,10 +6251,10 @@
href = ref.href;
title = ref.title;
}
// We found the end of the link, and know for a fact it's a valid link;
// so all that's left to do is to call tokenizer.
if (!silent) {
state.pos = labelStart;
state.posMax = labelEnd;
@@ -6375,10 +6375,10 @@
href = ref.href;
title = ref.title;
}
// We found the end of the link, and know for a fact it's a valid link;
// so all that's left to do is to call tokenizer.
if (!silent) {
content = state.src.slice(labelStart, labelEnd);
state.md.inline.parse(content, state.md, state.env, tokens = []);
@@ -6547,7 +6547,7 @@
// markers belong to same delimiter run if:
// - they have adjacent tokens
// - AND markers are the same
if (delimiters[headerIdx].marker !== closer.marker || lastTokenIdx !== closer.token - 1) {
headerIdx = closerIdx;
}
@@ -6555,7 +6555,7 @@
// Length is only used for emphasis-specific "rule of 3",
// if it's not defined (in strikethrough or 3rd party plugins),
// we can default it to 0 to disable those checks.
closer.length = closer.length || 0;
if (!closer.close) continue;
// Previously calculated lower bounds (previous fails)
@@ -6574,12 +6574,12 @@
if (opener.open && opener.end < 0) {
isOddMatch = false;
// from spec:
// If one of the delimiters can both open and close emphasis, then the
// sum of the lengths of the delimiter runs containing the opening and
// closing delimiters must not be a multiple of 3 unless both lengths
// are multiples of 3.
if (opener.close || closer.open) {
if ((opener.length + closer.length) % 3 === 0) {
if (opener.length % 3 !== 0 || closer.length % 3 !== 0) {
@@ -6678,7 +6678,7 @@
this.linkLevel = 0;
}
// Flush pending text
StateInline.prototype.pushPending = function() {
var token$1 = new token("text", "", 0);
token$1.content = this.pending;
@@ -6689,7 +6689,7 @@
};
// Push new token to "stream".
// If pending text exists - flush it as text token
StateInline.prototype.push = function(type, tag, nesting) {
if (this.pending) {
this.pushPending();
@@ -6718,10 +6718,10 @@
};
// Scan a sequence of emphasis-like markers, and determine whether
// it can start an emphasis sequence or end an emphasis sequence.
// - start - position to scan from (it should point at a valid marker);
// - canSplitWord - determine if these markers can be found inside a word
StateInline.prototype.scanDelims = function(start, canSplitWord) {
var pos = start, lastChar, nextChar, count, can_open, can_close, isLastWhiteSpace, isLastPunctChar, isNextWhiteSpace, isNextPunctChar, left_flanking = true, right_flanking = true, max = this.posMax, marker = this.src.charCodeAt(start);
// treat beginning of the line as a whitespace
@@ -6771,10 +6771,10 @@
var _rules = [ [ "text", text ], [ "linkify", linkify ], [ "newline", newline ], [ "escape", _escape ], [ "backticks", backticks ], [ "strikethrough", strikethrough.tokenize ], [ "emphasis", emphasis.tokenize ], [ "link", link ], [ "image", image ], [ "autolink", autolink ], [ "html_inline", html_inline ], [ "entity", entity ] ];
// `rule2` ruleset was created specifically for emphasis/strikethrough
// post-processing and may be changed in the future.
// Don't use this for anything except pairs (plugins working with `balance_pairs`).
var _rules2 = [ [ "balance_pairs", balance_pairs ], [ "strikethrough", strikethrough.postProcess ], [ "emphasis", emphasis.postProcess ],
var _rules2 = [ [ "balance_pairs", balance_pairs ], [ "strikethrough", strikethrough.postProcess ], [ "emphasis", emphasis.postProcess ],
// rules for pairs separate '**' into its own text tokens, which may be left unused,
// rule below merges unused segments back with the rest of the text
[ "fragments_join", fragments_join ] ];
@@ -6802,7 +6802,7 @@
}
// Skip single token by running all rules in validation mode;
// returns `true` if any rule reported success
ParserInline.prototype.skipToken = function(state) {
var ok, i, pos = state.pos, rules = this.ruler.getRules(""), len = rules.length, maxNesting = state.md.options.maxNesting, cache = state.cache;
if (typeof cache[pos] !== "undefined") {
@@ -6837,7 +6837,7 @@
cache[pos] = state.pos;
};
// Generate tokens for input range
ParserInline.prototype.tokenize = function(state) {
var ok, i, rules = this.ruler.getRules(""), len = rules.length, end = state.posMax, maxNesting = state.md.options.maxNesting;
while (state.pos < end) {
@@ -6928,11 +6928,11 @@
re.src_xn = "xn--[a-z0-9\\-]{1,59}";
// More to read about domain names
// http://serverfault.com/questions/638260/
re.src_domain_root =
re.src_domain_root =
// Allow letters & digits (http://test1)
"(?:" + re.src_xn + "|" + re.src_pseudo_letter + "{1,63}" + ")";
re.src_domain = "(?:" + re.src_xn + "|" + "(?:" + re.src_pseudo_letter + ")" + "|" + "(?:" + re.src_pseudo_letter + "(?:-|" + re.src_pseudo_letter + "){0,61}" + re.src_pseudo_letter + ")" + ")";
re.src_host = "(?:" +
re.src_host = "(?:" +
// Don't need IP check, because digits are already allowed in normal domain names
// src_ip4 +
// '|' +
@@ -6949,11 +6949,11 @@
// Rude test fuzzy links by host, for quick deny
re.tpl_host_fuzzy_test = "localhost|www\\.|\\.\\d{1,3}\\.|(?:\\.(?:%TLDS%)(?:" + re.src_ZPCc + "|>|$))";
re.tpl_email_fuzzy = "(^|" + text_separators + '|"|\\(|' + re.src_ZCc + ")" + "(" + re.src_email_name + "@" + re.tpl_host_fuzzy_strict + ")";
re.tpl_link_fuzzy =
re.tpl_link_fuzzy =
// Fuzzy link can't be prepended with .:/\- and non punctuation.
// but can start with > (markdown blockquote)
"(^|(?![.:/\\-_@])(?:[$+<=>^`|\uff5c]|" + re.src_ZPCc + "))" + "((?![$+<=>^`|\uff5c])" + re.tpl_host_port_fuzzy_strict + re.src_path + ")";
re.tpl_link_no_ip_fuzzy =
re.tpl_link_no_ip_fuzzy =
// Fuzzy link can't be prepended with .:/\- and non punctuation.
// but can start with > (markdown blockquote)
"(^|(?![.:/\\-_@])(?:[$+<=>^`|\uff5c]|" + re.src_ZPCc + "))" + "((?![$+<=>^`|\uff5c])" + re.tpl_host_port_no_ip_fuzzy_strict + re.src_path + ")";
@@ -6962,7 +6962,7 @@
////////////////////////////////////////////////////////////////////////////////
// Helpers
// Merge objects
function assign(obj /*from1, from2, from3, ...*/) {
var sources = Array.prototype.slice.call(arguments, 1);
sources.forEach((function(source) {
@@ -7025,7 +7025,7 @@
var tail = text.slice(pos);
if (!self.re.no_http) {
// compile lazily, because "host"-containing variables can change on tlds update.
self.re.no_http = new RegExp("^" + self.re.src_auth +
self.re.no_http = new RegExp("^" + self.re.src_auth +
// Don't allow single-level domains, because of false positives like '//test'
// with code comments
"(?:localhost|(?:(?:" + self.re.src_domain + ")\\.)+" + self.re.src_domain_root + ")" + self.re.src_port + self.re.src_host_terminator + self.re.src_path, "i");
@@ -7082,7 +7082,7 @@
};
}
// Schemas compiler. Build regexps.
function compile(self) {
// Load & clone RE patterns.
var re$1 = self.re = re(self.__opts__);
@@ -7101,9 +7101,9 @@
re$1.link_fuzzy = RegExp(untpl(re$1.tpl_link_fuzzy), "i");
re$1.link_no_ip_fuzzy = RegExp(untpl(re$1.tpl_link_no_ip_fuzzy), "i");
re$1.host_fuzzy_test = RegExp(untpl(re$1.tpl_host_fuzzy_test), "i");
// Compile each schema
var aliases = [];
self.__compiled__ = {};
// Reset compiled data
@@ -7144,9 +7144,9 @@
}
schemaError(name, val);
}));
// Compile postponed aliases
aliases.forEach((function(alias) {
if (!self.__compiled__[self.__schemas__[alias]]) {
// Silently fail on missed schemas to avoid errons on disable.
@@ -7156,16 +7156,16 @@
self.__compiled__[alias].validate = self.__compiled__[self.__schemas__[alias]].validate;
self.__compiled__[alias].normalize = self.__compiled__[self.__schemas__[alias]].normalize;
}));
// Fake record for guessed links
self.__compiled__[""] = {
validate: null,
normalize: createNormalizer()
};
// Build schema condition
var slist = Object.keys(self.__compiled__).filter((function(name) {
// Filter disabled & fake schemas
return name.length > 0 && self.__compiled__[name];
@@ -7175,9 +7175,9 @@
self.re.schema_search = RegExp("(^|(?!_)(?:[><\uff5c]|" + re$1.src_ZPCc + "))(" + slist + ")", "ig");
self.re.schema_at_start = RegExp("^" + self.re.schema_search.source, "i");
self.re.pretest = RegExp("(" + self.re.schema_test.source + ")|(" + self.re.host_fuzzy_test.source + ")|@", "i");
// Cleanup
resetScanCache(self);
}
/**
@@ -7673,7 +7673,7 @@
* @returns {String} The resulting string of Unicode symbols.
*/ function decode(input) {
// Don't use UCS-2
var output = [], inputLength = input.length, out, i = 0, n = initialN, bias = initialBias, basic, j, index, oldi, w, k, digit, t,
var output = [], inputLength = input.length, out, i = 0, n = initialN, bias = initialBias, basic, j, index, oldi, w, k, digit, t,
/** Cached calculation results */
baseMinusT;
// Handle the basic code points: let `basic` be the number of input code
@@ -7738,9 +7738,9 @@
* @param {String} input The string of Unicode symbols.
* @returns {String} The resulting Punycode string of ASCII-only symbols.
*/ function encode(input) {
var n, delta, handledCPCount, basicLength, bias, j, m, q, k, t, currentValue, output = [],
var n, delta, handledCPCount, basicLength, bias, j, m, q, k, t, currentValue, output = [],
/** `inputLength` will hold the number of code points in `input`. */
inputLength,
inputLength,
/** Cached calculation results */
handledCPCountPlusOne, baseMinusT, qMinusT;
// Convert the input in UCS-2 to Unicode
@@ -7993,13 +7993,13 @@
commonmark: commonmark
};
////////////////////////////////////////////////////////////////////////////////
// This validator can prohibit more than really needed to prevent XSS. It's a
// tradeoff to keep code simple and to be secure by default.
// If you need different setup - override validator method as you wish. Or
// replace it with dummy function and use external sanitizer.
var BAD_PROTO_RE = /^(vbscript|javascript|file|data):/;
var GOOD_DATA_RE = /^data:image\/(gif|png|jpeg|webp);/;
function validateLink(url) {

6
src/interface/web/assets/org.min.js → src/khoj/interface/web/assets/org.min.js vendored
View File

@@ -614,8 +614,10 @@ var Org = (function () {
var notBlankNextToken = this.lexer.peekNextToken();
if (blankToken && !notBlankNextToken.isListElement())
this.lexer.pushToken(blankToken); // Recover blank token only when next line is not listElement.
if (notBlankNextToken.indentation <= rootIndentation)
break; // end of the list
// End of the list if hit less indented line or end of directive
if (notBlankNextToken.indentation <= rootIndentation ||
(notBlankNextToken.type === Lexer.tokens.directive && notBlankNextToken.endDirective))
break;
var element = this.parseElement(); // recursive
if (element)

5
src/khoj/interface/web/assets/pico.min.css vendored Normal file
View File

File diff suppressed because one or more lines are too long

171
src/khoj/interface/web/base_config.html Normal file
View File

@@ -0,0 +1,171 @@
<!DOCTYPE html>
<html data-theme="light">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0 maximum-scale=1.0">
<link rel="icon" type="image/png" sizes="128x128" href="/static/assets/icons/favicon-128x128.png">
<title>Khoj - Settings</title>
<link rel="stylesheet" href="/static/assets/pico.min.css">
<link rel="stylesheet" href="/static/assets/khoj.css">
</head>
<body class="khoj-configure">
<div class="khoj-header-wrapper">
<div class="filler"></div>
<div class="khoj-header">
<a class="khoj-logo" href="https://lantern.khoj.dev" target="_blank">
<img class="khoj-logo" src="/static/assets/icons/khoj-logo-sideways.svg" alt="Khoj"></img>
</a>
<nav class="khoj-nav">
<a class="khoj-nav" href="/chat">Chat</a>
<a class="khoj-nav" href="/">Search</a>
<a class="khoj-nav khoj-nav-selected" href="/config">Settings</a>
</nav>
</div>
<div class="filler"></div>
</div>
<div class=”content”>
{% block content %}
{% endblock %}
</div>
</body>
<style>
html, body {
width: 100%;
margin: 0;
padding: 0;
}
@media only screen and (max-width: 600px) {
body {
display: grid;
grid-template-columns: 1fr;
grid-template-rows: 1fr auto auto auto minmax(80px, 100%);
}
body > * {
grid-column: 1;
}
div.filler {
display: none;
}
body.khoj-configure {
padding: 0;
}
}
img.khoj-logo {
max-width: none!important;
}
div.khoj-header-wrapper{
display: grid;
grid-template-columns: 1fr min(70vw, 100%) 1fr;
}
.page {
display: grid;
grid-auto-flow: row;
gap: 32px;
}
.section {
display: grid;
justify-self: center;
}
.section-title {
margin: 0;
padding: 0 0 16px 0;
font-size: 32;
font-weight: normal;
}
.section-cards {
display: grid;
grid-template-columns: repeat(2, 1fr);
gap: 16px;
justify-items: start;
}
.card {
display: grid;
grid-template-rows: repeat(3, 1fr);
gap: 8px;
padding: 24px 16px;
width: 320px;
height: 180px;
background: white;
border: 1px solid rgb(229, 229, 229);
border-radius: 4px;
box-shadow: 0px 1px 3px 0px rgba(0,0,0,0.1),0px 1px 2px -1px rgba(0,0,0,0.1);
overflow: hidden;
}
.card-title-row {
display: grid;
grid-template-columns: auto 1fr;
padding: 0;
gap: 12px;
}
.card-icon {
width: 40px;
height: 40px;
}
.card-title {
font-size: 20px;
font-weight: normal;
margin: 0;
padding: 0;
align-self: center;
}
.card-title-text {
vertical-align: middle;
}
.card-description {
margin: 0;
color: grey;
font-size: 16px;
}
.card-button-row {
display: grid;
grid-template-columns: auto;
text-align: right;
}
.card-button {
border: none;
font-weight: bold;
color: rgb(64,64,64);
background: transparent;
font-size: 16px;
cursor: pointer;
margin: 0;
padding: 0;
height: 32px;
text-align: right;
}
.primary-button {
border: none;
color: white;
padding: 15px 32px;
text-align: center;
text-decoration: none;
display: inline-block;
font-size: 16px;
}
button.card-button {
color: rgb(255, 136, 136);
background: transparent;
font-size: 16px;
cursor: pointer;
margin: 0;
padding: 0;
height: 32px;
text-align: right;
text-align: left;
}
img.configured-icon {
max-width: 16px;
}
@media screen and (max-width: 600px) {
.section-cards {
grid-template-columns: 1fr;
}
}
</style>
</html>

27
src/khoj/interface/web/base_processor_integration.html Normal file
View File

@@ -0,0 +1,27 @@
<!DOCTYPE html>
<html>
<head>
<title>Khoj: Processor Settings</title>
<link rel=”stylesheet” href=”static/styles.css”>
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/@picocss/pico@1/css/pico.min.css">
</head>
<body class="data-integration">
<header class=”header”>
<h1>Configure your processor integrations for Khoj</h1>
</header>
<a href="/config">Go back</a>
<div class=”content”>
{% block content %}
{% endblock %}
</div>
<footer class=”footer”>
</footer>
</body>
<style>
body.data-integration {
padding: 0 10%
}
</style>
</html>

463
src/khoj/interface/web/chat.html Normal file
View File

@@ -0,0 +1,463 @@
<html>
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0 maximum-scale=1.0">
<title>Khoj - Chat</title>
<link rel="icon" type="image/png" sizes="128x128" href="/static/assets/icons/favicon-128x128.png">
<link rel="manifest" href="/static/khoj_chat.webmanifest">
<link rel="stylesheet" href="/static/assets/khoj.css">
</head>
<script>
function formatDate(date) {
// Format date in HH:MM, DD MMM YYYY format
let time_string = date.toLocaleTimeString('en-IN', { hour: '2-digit', minute: '2-digit', hour12: false });
let date_string = date.toLocaleString('en-IN', { year: 'numeric', month: 'short', day: '2-digit'}).replaceAll('-', ' ');
return `${time_string}, ${date_string}`;
}
function generateReference(reference, index) {
// Escape reference for HTML rendering
let escaped_ref = reference.replaceAll('"', '&quot;');
// Generate HTML for Chat Reference
return `<sup><abbr title="${escaped_ref}" tabindex="0">${index}</abbr></sup>`;
}
function renderMessage(message, by, dt=null) {
let message_time = formatDate(dt ?? new Date());
let by_name = by == "khoj" ? "🏮 Khoj" : "🤔 You";
// Generate HTML for Chat Message and Append to Chat Body
document.getElementById("chat-body").innerHTML += `
<div data-meta="${by_name} at ${message_time}" class="chat-message ${by}">
<div class="chat-message-text ${by}">${message}</div>
</div>
`;
// Scroll to bottom of chat-body element
document.getElementById("chat-body").scrollTop = document.getElementById("chat-body").scrollHeight;
}
function renderMessageWithReference(message, by, context=null, dt=null) {
let references = '';
if (context) {
references = context
.map((reference, index) => generateReference(reference, index))
.join("<sup>,</sup>");
}
renderMessage(message+references, by, dt);
}
function chat() {
// Extract required fields for search from form
let query = document.getElementById("chat-input").value.trim();
let results_count = localStorage.getItem("khojResultsCount") || 5;
console.log(`Query: ${query}`);
// Short circuit on empty query
if (query.length === 0)
return;
// Add message by user to chat body
renderMessage(query, "you");
document.getElementById("chat-input").value = "";
// Generate backend API URL to execute query
let url = `/api/chat?q=${encodeURIComponent(query)}&n=${results_count}&client=web&stream=true`;
let chat_body = document.getElementById("chat-body");
let new_response = document.createElement("div");
new_response.classList.add("chat-message", "khoj");
new_response.attributes["data-meta"] = "🏮 Khoj at " + formatDate(new Date());
chat_body.appendChild(new_response);
let new_response_text = document.createElement("div");
new_response_text.classList.add("chat-message-text", "khoj");
new_response.appendChild(new_response_text);
// Temporary status message to indicate that Khoj is thinking
new_response_text.innerHTML = "🤔";
document.getElementById("chat-body").scrollTop = document.getElementById("chat-body").scrollHeight;
// Call specified Khoj API which returns a streamed response of type text/plain
fetch(url)
.then(response => {
const reader = response.body.getReader();
const decoder = new TextDecoder();
function readStream() {
reader.read().then(({ done, value }) => {
if (done) {
return;
}
// Decode message chunk from stream
const chunk = decoder.decode(value, { stream: true });
if (chunk.includes("### compiled references:")) {
const additionalResponse = chunk.split("### compiled references:")[0];
new_response_text.innerHTML += additionalResponse;
const rawReference = chunk.split("### compiled references:")[1];
const rawReferenceAsJson = JSON.parse(rawReference);
let polishedReference = rawReferenceAsJson.map((reference, index) => generateReference(reference, index))
.join("<sup>,</sup>");
new_response_text.innerHTML += polishedReference;
document.getElementById("chat-body").scrollTop = document.getElementById("chat-body").scrollHeight;
} else {
// Display response from Khoj
if (new_response_text.innerHTML === "🤔") {
// Clear temporary status message
new_response_text.innerHTML = "";
}
new_response_text.innerHTML += chunk;
readStream();
}
// Scroll to bottom of chat window as chat response is streamed
document.getElementById("chat-body").scrollTop = document.getElementById("chat-body").scrollHeight;
});
}
readStream();
});
}
function incrementalChat(event) {
// Send chat message on 'Enter'
if (event.key === 'Enter') {
chat();
}
}
window.onload = function () {
fetch('/api/chat/history?client=web')
.then(response => response.json())
.then(data => {
if (data.detail) {
// If the server returns a 500 error with detail, render it as a message.
renderMessage("Hi 👋🏾, to get started <br/>1. Get your <a class='inline-chat-link' href='https://platform.openai.com/account/api-keys'>OpenAI API key</a><br/>2. Save it in the Khoj <a class='inline-chat-link' href='/config/processor/conversation'>chat settings</a> <br/>3. Click Configure on the Khoj <a class='inline-chat-link' href='/config'>settings page</a>", "khoj");
// Disable chat input field and update placeholder text
document.getElementById("chat-input").setAttribute("disabled", "disabled");
document.getElementById("chat-input").setAttribute("placeholder", "Configure Khoj to enable chat");
} else {
// Set welcome message on load
renderMessage("Hey 👋🏾, what's up?", "khoj");
}
return data.response;
})
.then(response => {
// Render conversation history, if any
response.forEach(chat_log => {
renderMessageWithReference(chat_log.message, chat_log.by, chat_log.context, new Date(chat_log.created));
});
})
.catch(err => {
return;
});
// Fill query field with value passed in URL query parameters, if any.
var query_via_url = new URLSearchParams(window.location.search).get("q");
if (query_via_url) {
document.getElementById("chat-input").value = query_via_url;
chat();
}
}
</script>
<body>
<!--Add Header Logo and Nav Pane-->
<div class="khoj-header">
{% if demo %}
<!-- Banner linking to https://khoj.dev -->
<div class="khoj-banner-container">
<a class="khoj-banner" href="https://khoj.dev" target="_blank">
<p id="khoj-banner" class="khoj-banner">
Enroll in Khoj cloud to get your own Github assistant
</p>
</a>
<input type="text" id="khoj-banner-email" placeholder="email" class="khoj-banner-email"></input>
<button id="khoj-banner-submit" class="khoj-banner-button">Submit</button>
</div>
{% endif %}
{% if demo %}
<a class="khoj-logo" href="https://khoj.dev" target="_blank">
<img class="khoj-logo" src="/static/assets/icons/khoj-logo-sideways.svg" alt="Khoj"></img>
</a>
{% else %}
<a class="khoj-logo" href="/">
<img class="khoj-logo" src="/static/assets/icons/khoj-logo-sideways.svg" alt="Khoj"></img>
</a>
{% endif %}
<nav class="khoj-nav">
<a class="khoj-nav khoj-nav-selected" href="/chat">Chat</a>
<a class="khoj-nav" href="/">Search</a>
{% if not demo %}
<a class="khoj-nav" href="/config">Settings</a>
{% endif %}
</nav>
</div>
<!-- Chat Body -->
<div id="chat-body"></div>
<!-- Chat Footer -->
<div id="chat-footer">
<input type="text" id="chat-input" class="option" onkeyup=incrementalChat(event) autofocus="autofocus" placeholder="What is the meaning of life?">
</div>
</body>
<style>
html, body {
height: 100%;
width: 100%;
padding: 0px;
margin: 0px;
}
body {
display: grid;
background: #f8fafc;
color: #475569;
text-align: center;
font-family: roboto, karma, segoe ui, sans-serif;
font-size: 20px;
font-weight: 300;
line-height: 1.5em;
}
body > * {
padding: 10px;
margin: 10px;
}
#chat-body {
font-size: medium;
margin: 0px;
line-height: 20px;
overflow-y: scroll; /* Make chat body scroll to see history */
}
/* add chat metatdata to bottom of bubble */
.chat-message::after {
content: attr(data-meta);
display: block;
font-size: x-small;
color: #475569;
margin: -8px 4px 0 -5px;
}
/* move message by khoj to left */
.chat-message.khoj {
margin-left: auto;
text-align: left;
}
/* move message by you to right */
.chat-message.you {
margin-right: auto;
text-align: right;
}
/* basic style chat message text */
.chat-message-text {
margin: 10px;
border-radius: 10px;
padding: 10px;
position: relative;
display: inline-block;
max-width: 80%;
text-align: left;
}
/* color chat bubble by khoj blue */
.chat-message-text.khoj {
color: var(--primary-inverse);
background: var(--primary);
margin-left: auto;
white-space: pre-line;
}
/* add left protrusion to khoj chat bubble */
.chat-message-text.khoj:after {
content: '';
position: absolute;
bottom: -2px;
left: -7px;
border: 10px solid transparent;
border-top-color: var(--primary);
border-bottom: 0;
transform: rotate(-60deg);
}
/* color chat bubble by you dark grey */
.chat-message-text.you {
color: #f8fafc;
background: #475569;
margin-right: auto;
}
/* add right protrusion to you chat bubble */
.chat-message-text.you:after {
content: '';
position: absolute;
top: 91%;
right: -2px;
border: 10px solid transparent;
border-left-color: #475569;
border-right: 0;
margin-top: -10px;
transform: rotate(-60deg)
}
#chat-footer {
padding: 0;
display: grid;
grid-template-columns: minmax(70px, 100%);
grid-column-gap: 10px;
grid-row-gap: 10px;
}
#chat-footer > * {
padding: 15px;
border-radius: 5px;
border: 1px solid #475569;
background: #f9fafc
}
.option:hover {
box-shadow: 0 0 11px #aaa;
}
#chat-input {
font-size: medium;
}
a.inline-chat-link {
color: #475569;
text-decoration: none;
border-bottom: 1px dotted #475569;
}
@media (pointer: coarse), (hover: none) {
abbr[title] {
position: relative;
padding-left: 4px; /* space references out to ease tapping */
}
abbr[title]:focus:after {
content: attr(title);
/* position tooltip */
position: absolute;
left: 16px; /* open tooltip to right of ref link, instead of on top of it */
width: auto;
z-index: 1; /* show tooltip above chat messages */
/* style tooltip */
background-color: #aaa;
color: #f8fafc;
border-radius: 2px;
box-shadow: 1px 1px 4px 0 rgba(0, 0, 0, 0.4);
font-size: 14px;
padding: 2px 4px;
}
}
@media only screen and (max-width: 600px) {
body {
grid-template-columns: 1fr;
grid-template-rows: auto minmax(80px, 100%) auto;
}
body > * {
grid-column: 1;
}
#chat-footer {
padding: 0;
margin: 4px;
grid-template-columns: auto;
}
a.khoj-banner {
display: block;
}
}
@media only screen and (min-width: 600px) {
body {
grid-template-columns: auto min(70vw, 100%) auto;
grid-template-rows: auto minmax(80px, 100%) auto;
}
body > * {
grid-column: 2;
}
}
div.khoj-banner-container {
background: linear-gradient(-45deg, #FFC107, #FF9800, #FF5722, #FF9800, #FFC107);
background-size: 400% 400%;
animation: gradient 15s ease infinite;
text-align: center;
padding: 10px;
}
@keyframes gradient {
0% {
background-position: 0% 50%;
}
50% {
background-position: 100% 50%;
}
100% {
background-position: 0% 50%;
}
}
a.khoj-banner {
color: black;
}
a.khoj-logo {
text-align: center;
}
p.khoj-banner {
margin: 0;
padding: 10px;
}
button#khoj-banner-submit,
input#khoj-banner-email {
padding: 10px;
border-radius: 5px;
border: 1px solid #475569;
background: #f9fafc;
}
button#khoj-banner-submit:hover,
input#khoj-banner-email:hover {
box-shadow: 0 0 11px #aaa;
}
p#khoj-banner {
display: inline;
}
a.khoj-banner {
color: black;
text-decoration: none;
}
</style>
<script>
var khojBannerSubmit = document.getElementById("khoj-banner-submit");
khojBannerSubmit?.addEventListener("click", function(event) {
event.preventDefault();
var email = document.getElementById("khoj-banner-email").value;
fetch("https://lantern.khoj.dev/beta/users/", {
method: "POST",
body: JSON.stringify({
email: email
}),
headers: {
"Content-Type": "application/json"
}
}).then(function(response) {
return response.json();
}).then(function(data) {
console.log(data);
if (data.user != null) {
document.getElementById("khoj-banner").innerHTML = "Thanks for signing up. We'll be in touch soon! 🚀";
document.getElementById("khoj-banner-submit").remove();
} else {
document.getElementById("khoj-banner").innerHTML = "There was an error signing up. Please contact team@khoj.dev";
}
}).catch(function(error) {
console.log(error);
document.getElementById("khoj-banner").innerHTML = "There was an error signing up. Please contact team@khoj.dev";
});
});
</script>
</html>

329
src/khoj/interface/web/config.html Normal file
View File

@@ -0,0 +1,329 @@
{% extends "base_config.html" %}
{% block content %}
<div class="page">
<div class="section">
<h2 class="section-title">Plugins</h2>
<div class="section-cards">
<div class="card">
<div class="card-title-row">
<img class="card-icon" src="/static/assets/icons/github.svg" alt="Github">
<h3 class="card-title">
Github
{% if current_config.content_type.github %}
<img id="configured-icon-github" class="configured-icon" src="/static/assets/icons/confirm-icon.svg" alt="Configured">
{% endif %}
</h3>
</div>
<div class="card-description-row">
<p class="card-description">Set repositories for Khoj to index</p>
</div>
<div class="card-action-row">
<a class="card-button" href="/config/content_type/github">
{% if current_config.content_type.github %}
Update
{% else %}
Setup
{% endif %}
<svg xmlns="http://www.w3.org/2000/svg" width="1em" height="1em" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M5 12h14M12 5l7 7-7 7"></path></svg>
</a>
</div>
{% if current_config.content_type.github %}
<div id="clear-github" class="card-action-row">
<button class="card-button" onclick="clearContentType('github')">
Disable
</button>
</div>
{% endif %}
</div>
<div class="card">
<div class="card-title-row">
<img class="card-icon" src="/static/assets/icons/notion.svg" alt="Notion">
<h3 class="card-title">
Notion
{% if current_config.content_type.notion %}
<img id="configured-icon-notion" class="configured-icon" src="/static/assets/icons/confirm-icon.svg" alt="Configured">
{% endif %}
</h3>
</div>
<div class="card-description-row">
<p class="card-description">Configure your settings from Notion</p>
</div>
<div class="card-action-row">
<a class="card-button" href="/config/content_type/notion">
{% if current_config.content_type.content %}
Update
{% else %}
Setup
{% endif %}
<svg xmlns="http://www.w3.org/2000/svg" width="1em" height="1em" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M5 12h14M12 5l7 7-7 7"></path></svg>
</a>
</div>
{% if current_config.content_type.notion %}
<div id="clear-notion" class="card-action-row">
<button class="card-button" onclick="clearContentType('notion')">
Disable
</button>
</div>
{% endif %}
</div>
<div class="card">
<div class="card-title-row">
<img class="card-icon" src="/static/assets/icons/markdown.svg" alt="markdown">
<h3 class="card-title">
Markdown
{% if current_config.content_type.markdown %}
<img id="configured-icon-markdown" class="configured-icon" src="/static/assets/icons/confirm-icon.svg" alt="Configured">
{% endif %}
</h3>
</div>
<div class="card-description-row">
<p class="card-description">Set markdown files for Khoj to index</p>
</div>
<div class="card-action-row">
<a class="card-button" href="/config/content_type/markdown">
{% if current_config.content_type.markdown %}
Update
{% else %}
Setup
{% endif %}
<svg xmlns="http://www.w3.org/2000/svg" width="1em" height="1em" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M5 12h14M12 5l7 7-7 7"></path></svg>
</a>
</div>
{% if current_config.content_type.markdown %}
<div id="clear-markdown" class="card-action-row">
<button class="card-button" onclick="clearContentType('markdown')">
Disable
</button>
</div>
{% endif %}
</div>
<div class="card">
<div class="card-title-row">
<img class="card-icon" src="/static/assets/icons/org.svg" alt="org">
<h3 class="card-title">
Org
{% if current_config.content_type.org %}
<img id="configured-icon-org" class="configured-icon" src="/static/assets/icons/confirm-icon.svg" alt="Configured">
{% endif %}
</h3>
</div>
<div class="card-description-row">
<p class="card-description">Set org files for Khoj to index</p>
</div>
<div class="card-action-row">
<a class="card-button" href="/config/content_type/org">
{% if current_config.content_type.org %}
Update
{% else %}
Setup
{% endif %}
<svg xmlns="http://www.w3.org/2000/svg" width="1em" height="1em" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M5 12h14M12 5l7 7-7 7"></path></svg>
</a>
</div>
{% if current_config.content_type.org %}
<div id="clear-org" class="card-action-row">
<button class="card-button" onclick="clearContentType('org')">
Disable
</button>
</div>
{% endif %}
</div>
<div class="card">
<div class="card-title-row">
<img class="card-icon" src="/static/assets/icons/pdf.svg" alt="PDF">
<h3 class="card-title">
PDF
{% if current_config.content_type.pdf %}
<img id="configured-icon-pdf" class="configured-icon" src="/static/assets/icons/confirm-icon.svg" alt="Configured">
{% endif %}
</h3>
</div>
<div class="card-description-row">
<p class="card-description">Set PDF files for Khoj to index</p>
</div>
<div class="card-action-row">
<a class="card-button" href="/config/content_type/pdf">
{% if current_config.content_type.pdf %}
Update
{% else %}
Setup
{% endif %}
<svg xmlns="http://www.w3.org/2000/svg" width="1em" height="1em" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M5 12h14M12 5l7 7-7 7"></path></svg>
</a>
</div>
{% if current_config.content_type.pdf %}
<div id="clear-pdf" class="card-action-row">
<button class="card-button" onclick="clearContentType('pdf')">
Disable
</button>
</div>
{% endif %}
</div>
</div>
</div>
<div class="section">
<h2 class="section-title">Features</h2>
<div class="section-cards">
<div class="card">
<div class="card-title-row">
<img class="card-icon" src="/static/assets/icons/chat.svg" alt="Chat">
<h3 class="card-title">
Chat
{% if current_config.processor and current_config.processor.conversation %}
<img id="configured-icon-conversation-processor" class="configured-icon" src="/static/assets/icons/confirm-icon.svg" alt="Configured">
{% endif %}
</h3>
</div>
<div class="card-description-row">
<p class="card-description">Setup Khoj Chat with OpenAI</p>
</div>
<div class="card-action-row">
<a class="card-button" href="/config/processor/conversation">
{% if current_config.processor and current_config.processor.conversation %}
Update
{% else %}
Setup
{% endif %}
<svg xmlns="http://www.w3.org/2000/svg" width="1em" height="1em" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M5 12h14M12 5l7 7-7 7"></path></svg>
</a>
</div>
{% if current_config.processor and current_config.processor.conversation %}
<div id="clear-conversation" class="card-action-row">
<button class="card-button" onclick="clearConversationProcessor()">
Disable
</button>
</div>
{% endif %}
</div>
</div>
</div>
<div class="section">
<div id="results-count" title="Number of items to show in search and use for chat response">
<label for="results-count-slider">Results Count: <span id="results-count-value">5</span></label>
<input type="range" id="results-count-slider" name="results-count-slider" min="1" max="10" step="1" value="5">
</div>
<div id="status" style="display: none;"></div>
<button id="configure" type="submit" title="Update index with the latest changes">⚙️ Configure</button>
<button id="reinitialize" type="submit" title="Regenerate index from scratch">🔄 Reinitialize</button>
</div>
</div>
<script>
function clearContentType(content_type) {
const csrfToken = document.cookie.split('; ').find(row => row.startsWith('csrftoken'))?.split('=')[1];
fetch('/api/delete/config/data/content_type/' + content_type, {
method: 'POST',
headers: {
'Content-Type': 'application/json',
'X-CSRFToken': csrfToken
}
})
.then(response => response.json())
.then(data => {
if (data.status == "ok") {
var contentTypeClearButton = document.getElementById("clear-" + content_type);
contentTypeClearButton.style.display = "none";
var configuredIcon = document.getElementById("configured-icon-" + content_type);
configuredIcon.style.display = "none";
}
})
};
function clearConversationProcessor() {
const csrfToken = document.cookie.split('; ').find(row => row.startsWith('csrftoken'))?.split('=')[1];
fetch('/api/delete/config/data/processor/conversation', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
'X-CSRFToken': csrfToken
}
})
.then(response => response.json())
.then(data => {
if (data.status == "ok") {
var conversationClearButton = document.getElementById("clear-conversation");
conversationClearButton.style.display = "none";
var configuredIcon = document.getElementById("configured-icon-conversation-processor");
configuredIcon.style.display = "none";
}
})
};
var configure = document.getElementById("configure");
configure.addEventListener("click", function(event) {
event.preventDefault();
updateIndex(
force=false,
successText="Configured successfully!",
errorText="Unable to configure. Raise issue on Khoj <a href='https://github.com/khoj-ai/khoj/issues'>Github</a> or <a href='https://discord.gg/BDgyabRM6e'>Discord</a>.",
button=configure,
loadingText="Configuring...",
emoji="⚙️");
});
var reinitialize = document.getElementById("reinitialize");
reinitialize.addEventListener("click", function(event) {
event.preventDefault();
updateIndex(
force=true,
successText="Reinitialized successfully!",
errorText="Unable to reinitialize. Raise issue on Khoj <a href='https://github.com/khoj-ai/khoj/issues'>Github</a> or <a href='https://discord.gg/BDgyabRM6e'>Discord</a>.",
button=reinitialize,
loadingText="Reinitializing...",
emoji="🔄");
});
function updateIndex(force, successText, errorText, button, loadingText, emoji) {
const csrfToken = document.cookie.split('; ').find(row => row.startsWith('csrftoken'))?.split('=')[1];
button.disabled = true;
button.innerHTML = emoji + loadingText;
fetch('/api/update?&client=web&force=' + force, {
method: 'GET',
headers: {
'Content-Type': 'application/json',
'X-CSRFToken': csrfToken
}
})
.then(response => response.json())
.then(data => {
console.log('Success:', data);
if (data.detail != null) {
throw new Error(data.detail);
}
document.getElementById("status").innerHTML = emoji + successText;
document.getElementById("status").style.display = "block";
button.disabled = false;
button.innerHTML = '✅ Done!';
})
.catch((error) => {
console.error('Error:', error);
document.getElementById("status").innerHTML = emoji + errorText
document.getElementById("status").style.display = "block";
button.disabled = false;
button.innerHTML = '⚠️ Unsuccessful';
});
}
// Setup the results count slider
const resultsCountSlider = document.getElementById('results-count-slider');
const resultsCountValue = document.getElementById('results-count-value');
// Set the initial value of the slider
resultsCountValue.textContent = resultsCountSlider.value;
// Store the slider value in localStorage when it changes
resultsCountSlider.addEventListener('input', () => {
resultsCountValue.textContent = resultsCountSlider.value;
localStorage.setItem('khojResultsCount', resultsCountSlider.value);
});
// Get the slider value from localStorage on page load
const storedResultsCount = localStorage.getItem('khojResultsCount');
if (storedResultsCount) {
resultsCountSlider.value = storedResultsCount;
resultsCountValue.textContent = storedResultsCount;
}
</script>
{% endblock %}

170
src/khoj/interface/web/content_type_github_input.html Normal file
View File

@@ -0,0 +1,170 @@
{% extends "base_config.html" %}
{% block content %}
<div class="page">
<div class="section">
<h2 class="section-title">
<img class="card-icon" src="/static/assets/icons/github.svg" alt="Github">
<span class="card-title-text">Github</span>
</h2>
<form>
<table>
<tr>
<td>
<label for="pat-token">Personal Access Token</label>
</td>
<td>
<input type="text" id="pat-token" name="pat" value="{{ current_config['pat_token'] }}">
</td>
</tr>
</table>
<h4>Repositories</h4>
<div id="repositories" class="section-cards">
{% for repo in current_config['repos'] %}
<div class="card repo" id="repo-card-{{loop.index}}">
<label for="repo-owner">Repository Owner</label>
<input type="text" id="repo-owner-{{loop.index}}" name="repo_owner" value="{{ repo.owner }}">
<label for="repo-name">Repository Name</label>
<input type="text" id="repo-name-{{loop.index}}" name="repo_name" value="{{ repo.name}}">
<label for="repo-branch">Repository Branch</label>
<input type="text" id="repo-branch-{{loop.index}}" name="repo_branch" value="{{ repo.branch }}">
<button type="button"
class="remove-repo-button"
onclick="remove_repo({{loop.index}})"
id="remove-repo-button-{{loop.index}}">Remove Repository</button>
</div>
{% endfor %}
</div>
<button type="button" id="add-repository-button">Add Repository</button>
<table style="display: none;" >
<tr>
<td>
<label for="compressed-jsonl">Compressed JSONL (Output)</label>
</td>
<td>
<input type="text" id="compressed-jsonl" name="compressed-jsonl" value="{{ current_config['compressed_jsonl'] }}">
</td>
</tr>
<tr>
<td>
<label for="embeddings-file">Embeddings File (Output)</label>
</td>
<td>
<input type="text" id="embeddings-file" name="embeddings-file" value="{{ current_config['embeddings_file'] }}">
</td>
</tr>
</table>
<div class="section">
<div id="success" style="display: none;"></div>
<button id="submit" type="submit">Save</button>
</div>
</form>
</div>
</div>
<style>
div.repo {
width: 100%;
height: 100%;
grid-template-rows: none;
}
div#repositories {
margin-bottom: 12px;
}
button.remove-repo-button {
background-color: gainsboro;
}
</style>
<script>
const add_repo_button = document.getElementById("add-repository-button");
add_repo_button.addEventListener("click", function(event) {
event.preventDefault();
var repo = document.createElement("div");
repo.classList.add("card");
repo.classList.add("repo");
const id = Date.now();
repo.id = "repo-card-" + id;
repo.innerHTML = `
<label for="repo-owner">Repository Owner</label>
<input type="text" id="repo-owner" name="repo_owner">
<label for="repo-name">Repository Name</label>
<input type="text" id="repo-name" name="repo_name">
<label for="repo-branch">Repository Branch</label>
<input type="text" id="repo-branch" name="repo_branch">
<button type="button"
class="remove-repo-button"
onclick="remove_repo(${id})"
id="remove-repo-button-${id}">Remove Repository</button>
`;
document.getElementById("repositories").appendChild(repo);
})
function remove_repo(index) {
document.getElementById("repo-card-" + index).remove();
}
submit.addEventListener("click", function(event) {
event.preventDefault();
const compressed_jsonl = document.getElementById("compressed-jsonl").value;
const embeddings_file = document.getElementById("embeddings-file").value;
const pat_token = document.getElementById("pat-token").value;
if (pat_token == "") {
document.getElementById("success").innerHTML = "❌ Please enter a Personal Access Token.";
document.getElementById("success").style.display = "block";
return;
}
var cards = document.getElementById("repositories").getElementsByClassName("repo");
var repos = [];
for (var i = 0; i < cards.length; i++) {
var card = cards[i];
var owner = card.getElementsByTagName("input")[0].value;
var name = card.getElementsByTagName("input")[1].value;
var branch = card.getElementsByTagName("input")[2].value;
if (owner == "" || name == "" || branch == "") {
continue;
}
repos.push({
"owner": owner,
"name": name,
"branch": branch,
});
}
if (repos.length == 0) {
document.getElementById("success").innerHTML = "❌ Please add at least one repository.";
document.getElementById("success").style.display = "block";
return;
}
const csrfToken = document.cookie.split('; ').find(row => row.startsWith('csrftoken'))?.split('=')[1];
fetch('/api/config/data/content_type/github', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
'X-CSRFToken': csrfToken,
},
body: JSON.stringify({
"pat_token": pat_token,
"repos": repos,
"compressed_jsonl": compressed_jsonl,
"embeddings_file": embeddings_file,
})
})
.then(response => response.json())
.then(data => {
if (data["status"] == "ok") {
document.getElementById("success").innerHTML = "✅ Successfully updated. Go to your <a href='/config'>settings page</a> to complete setup.";
document.getElementById("success").style.display = "block";
} else {
document.getElementById("success").innerHTML = "⚠️ Failed to update settings.";
document.getElementById("success").style.display = "block";
}
})
});
</script>
{% endblock %}

167
src/khoj/interface/web/content_type_input.html Normal file
View File

@@ -0,0 +1,167 @@
{% extends "base_config.html" %}
{% block content %}
<div class="page">
<div class="section">
<h2 class="section-title">
<img class="card-icon" src="/static/assets/icons/{{ content_type }}.svg" alt="{{ content_type|capitalize }}">
<span class="card-title-text">{{ content_type|capitalize }}</span>
</h2>
<form id="config-form">
<table>
<tr>
<td>
<label for="input-files" title="Add a {{content_type}} file for Khoj to index">Files</label>
</td>
<td id="input-files-cell">
{% if current_config['input_files'] is none %}
<input type="text" id="input-files" name="input-files" placeholder="~\Documents\notes.{{content_type}}">
{% else %}
{% for input_file in current_config['input_files'] %}
<input type="text" id="input-files" name="input-files" value="{{ input_file }}" placeholder="~\Documents\notes.{{content_type}}">
{% endfor %}
{% endif %}
</td>
<td>
<button type="button" id="input-files-button">Add</button>
</td>
</tr>
<tr>
<td>
<label for="input-filter" title="Add a folder with {{content_type}} files for Khoj to index">Folders</label>
</td>
<td id="input-filter-cell">
{% if current_config['input_filter'] is none %}
<input type="text" id="input-filter" name="input-filter" placeholder="~/Documents/{{content_type}}">
{% else %}
{% for input_filter in current_config['input_filter'] %}
<input type="text" id="input-filter" name="input-filter" placeholder="~/Documents/{{content_type}}" value="{{ input_filter.split('/*')[0] }}">
{% endfor %}
{% endif %}
</td>
<td>
<button type="button" id="input-filter-button">Add</button>
</td>
</tr>
</table>
<table style="display: none;" >
<tr>
<td>
<label for="compressed-jsonl">Compressed JSONL (Output)</label>
</td>
<td>
<input type="text" id="compressed-jsonl" name="compressed-jsonl" value="{{ current_config['compressed_jsonl'] }}">
</td>
</tr>
<tr>
<td>
<label for="embeddings-file">Embeddings File (Output)</label>
</td>
<td>
<input type="text" id="embeddings-file" name="embeddings-file" value="{{ current_config['embeddings_file'] }}">
</td>
</tr>
<tr>
<td>
<label for="index-heading-entries">Index Heading Entries</label>
</td>
<td>
<input type="text" id="index-heading-entries" name="index-heading-entries" value="{{ current_config['index_heading_entries'] }}">
</td>
</tr>
</table>
<div class="section">
<div id="success" style="display: none;" ></div>
<button id="submit" type="submit">Save</button>
</div>
</form>
</div>
</div>
<script>
function addButtonEventListener(fieldName) {
var button = document.getElementById(fieldName + "-button");
button.addEventListener("click", function(event) {
var cell = document.getElementById(fieldName + "-cell");
var newInput = document.createElement("input");
newInput.setAttribute("type", "text");
newInput.setAttribute("name", fieldName);
cell.appendChild(newInput);
})
}
addButtonEventListener("input-files");
addButtonEventListener("input-filter");
function getValidInputNodes(nodes) {
var validNodes = [];
for (var i = 0; i < nodes.length; i++) {
const nodeValue = nodes[i].value;
if (nodeValue === "" || nodeValue === null || nodeValue === undefined || nodeValue === "None") {
continue;
}
validNodes.push(nodes[i]);
}
return validNodes;
}
submit.addEventListener("click", function(event) {
event.preventDefault();
let suffix = ""
if ('{{content_type}}' == "markdown")
suffix = "**/*.md"
else if ('{{content_type}}' == "org")
suffix = "**/*.org"
else if ('{{content_type}}' === "pdf")
suffix = "**/*.pdf"
var inputFileNodes = document.getElementsByName("input-files");
var input_files = getValidInputNodes(inputFileNodes).map(node => node.value);
var inputFilterNodes = document.getElementsByName("input-filter");
var input_filter = getValidInputNodes(inputFilterNodes).map(node => `${node.value}/${suffix}`);
if (input_files.length === 0 && input_filter.length === 0) {
alert("You must specify at least one input file or input filter.");
return;
}
if (input_files.length == 0) {
input_files = null;
}
if (input_filter.length == 0) {
input_filter = null;
}
var compressed_jsonl = document.getElementById("compressed-jsonl").value;
var embeddings_file = document.getElementById("embeddings-file").value;
var index_heading_entries = document.getElementById("index-heading-entries").value;
const csrfToken = document.cookie.split('; ').find(row => row.startsWith('csrftoken'))?.split('=')[1];
fetch('/api/config/data/content_type/{{ content_type }}', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
'X-CSRFToken': csrfToken
},
body: JSON.stringify({
"input_files": input_files,
"input_filter": input_filter,
"compressed_jsonl": compressed_jsonl,
"embeddings_file": embeddings_file,
"index_heading_entries": index_heading_entries
})
})
.then(response => response.json())
.then(data => {
if (data["status"] == "ok") {
document.getElementById("success").innerHTML = "✅ Successfully updated. Go to your <a href='/config'>settings page</a> to complete setup.";
document.getElementById("success").style.display = "block";
} else {
document.getElementById("success").innerHTML = "⚠️ Failed to update settings.";
document.getElementById("success").style.display = "block";
}
})
});
</script>
{% endblock %}

86
src/khoj/interface/web/content_type_notion_input.html Normal file
View File

@@ -0,0 +1,86 @@
{% extends "base_config.html" %}
{% block content %}
<div class="page">
<div class="section">
<h2 class="section-title">
<img class="card-icon" src="/static/assets/icons/notion.svg" alt="Notion">
<span class="card-title-text">Notion</span>
</h2>
<form>
<table>
<tr>
<td>
<label for="token">Token</label>
</td>
<td>
<input type="text" id="token" name="pat" value="{{ current_config['token'] }}">
</td>
</tr>
</table>
<table style="display: none;" >
<tr>
<td>
<label for="compressed-jsonl">Compressed JSONL (Output)</label>
</td>
<td>
<input type="text" id="compressed-jsonl" name="compressed-jsonl" value="{{ current_config['compressed_jsonl'] }}">
</td>
</tr>
<tr>
<td>
<label for="embeddings-file">Embeddings File (Output)</label>
</td>
<td>
<input type="text" id="embeddings-file" name="embeddings-file" value="{{ current_config['embeddings_file'] }}">
</td>
</tr>
</table>
<div class="section">
<div id="success" style="display: none;"></div>
<button id="submit" type="submit">Save</button>
</div>
</form>
</div>
</div>
<script>
const submit = document.getElementById("submit");
submit.addEventListener("click", function(event) {
event.preventDefault();
const compressed_jsonl = document.getElementById("compressed-jsonl").value;
const embeddings_file = document.getElementById("embeddings-file").value;
const token = document.getElementById("token").value;
if (token == "") {
document.getElementById("success").innerHTML = "❌ Please enter a Notion Token.";
document.getElementById("success").style.display = "block";
return;
}
const csrfToken = document.cookie.split('; ').find(row => row.startsWith('csrftoken'))?.split('=')[1];
fetch('/api/config/data/content_type/notion', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
'X-CSRFToken': csrfToken,
},
body: JSON.stringify({
"token": token,
"compressed_jsonl": compressed_jsonl,
"embeddings_file": embeddings_file,
})
})
.then(response => response.json())
.then(data => {
if (data["status"] == "ok") {
document.getElementById("success").innerHTML = "✅ Successfully updated. Go to your <a href='/config'>settings page</a> to complete setup.";
document.getElementById("success").style.display = "block";
} else {
document.getElementById("success").innerHTML = "⚠️ Failed to update settings.";
document.getElementById("success").style.display = "block";
}
})
});
</script>
{% endblock %}

517
src/khoj/interface/web/index.html Normal file
View File

@@ -0,0 +1,517 @@
<html>
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0 maximum-scale=1.0">
<title>Khoj - Search</title>
<link rel="icon" type="image/png" sizes="128x128" href="/static/assets/icons/favicon-128x128.png">
<link rel="manifest" href="/static/khoj.webmanifest">
<link rel="stylesheet" href="/static/assets/khoj.css">
</head>
<script type="text/javascript" src="/static/assets/org.min.js"></script>
<script type="text/javascript" src="/static/assets/markdown-it.min.js"></script>
<script>
function render_image(item) {
return `
<div class="results-image">
<a href="${item.entry}" class="image-link">
<img id=${item.score} src="${item.entry}?${Math.random()}"
title="Effective Score: ${item.score}, Meta: ${item.additional.metadata_score}, Image: ${item.additional.image_score}"
class="image">
</a>
</div>`;
}
function render_org(query, data, classPrefix="") {
return data.map(function (item) {
var orgParser = new Org.Parser();
var orgDocument = orgParser.parse(item.entry);
var orgHTMLDocument = orgDocument.convert(Org.ConverterHTML, { htmlClassPrefix: classPrefix });
return `<div class="results-org">` + orgHTMLDocument.toString() + `</div>`;
}).join("\n");
}
function render_markdown(query, data) {
var md = window.markdownit();
return data.map(function (item) {
let rendered = "";
if (item.additional.file.startsWith("http")) {
lines = item.entry.split("\n");
rendered = md.render(`${lines[0]}\t[*](${item.additional.file})\n${lines.slice(1).join("\n")}`);
}
else {
rendered = md.render(`${item.entry}`);
}
return `<div class="results-markdown">` + rendered + `</div>`;
}).join("\n");
}
function render_pdf(query, data) {
return data.map(function (item) {
let compiled_lines = item.additional.compiled.split("\n");
let filename = compiled_lines.shift();
let text_match = compiled_lines.join("\n")
return `<div class="results-pdf">` + `<h2>${filename}</h2>\n<p>${text_match}</p>` + `</div>`;
}).join("\n");
}
function render_multiple(query, data, type) {
let html = "";
data.forEach(item => {
if (item.additional.file.endsWith(".org")) {
html += render_org(query, [item], "org-");
} else if (
item.additional.file.endsWith(".md") ||
item.additional.file.endsWith(".markdown") ||
(item.additional.file.includes("issues") && item.additional.file.includes("github.com")) ||
(item.additional.file.includes("commit") && item.additional.file.includes("github.com"))
)
{
html += render_markdown(query, [item]);
} else if (item.additional.file.endsWith(".pdf")) {
html += render_pdf(query, [item]);
} else if (item.additional.file.includes("notion.so")) {
html += `<div class="results-notion">` + `<b><a href="${item.additional.file}">${item.additional.heading}</a></b>` + `<p>${item.entry}</p>` + `</div>`;
}
});
return html;
}
function render_results(data, query, type) {
let results = "";
if (type === "markdown") {
results = render_markdown(query, data);
} else if (type === "org") {
results = render_org(query, data, "org-");
} else if (type === "image") {
results = data.map(render_image).join('');
} else if (type === "pdf") {
results = render_pdf(query, data);
} else if (type === "github" || type === "all" || type === "notion") {
results = render_multiple(query, data, type);
} else {
results = data.map((item) => `<div class="results-plugin">` + `<p>${item.entry}</p>` + `</div>`).join("\n")
}
// Any POST rendering goes here.
let renderedResults = document.createElement("div");
renderedResults.id = `results-${type}`;
renderedResults.innerHTML = results;
// For all elements that are of type img in the results html and have a src with 'avatar' in the URL, add the class 'avatar'
// This is used to make the avatar images round
let images = renderedResults.querySelectorAll("img[src*='avatar']");
for (let i = 0; i < images.length; i++) {
images[i].classList.add("avatar");
}
return renderedResults.outerHTML;
}
function search(rerank=false) {
// Extract required fields for search from form
query = document.getElementById("query").value.trim();
type = document.getElementById("type").value;
searchHint = document.getElementById("info-hint");
results_count = localStorage.getItem("khojResultsCount") || 5;
console.log(`Query: ${query}, Type: ${type}, Results Count: ${results_count}`);
// Short circuit on empty query
if (query.length === 0) {
searchHint.style.display = "none";
return;
}
// If set query field in url query param on rerank
if (rerank)
setQueryFieldInUrl(query);
// Execute Search and Render Results
url = createRequestUrl(query, type, results_count || 5, rerank);
fetch(url)
.then(response => response.json())
.then(data => {
console.log(data);
document.getElementById("results").innerHTML = render_results(data, query, type);
});
setTimeout(() => { searchHint.style.display = "block"; }, 3000);
}
function incrementalSearch(event) {
type = document.getElementById("type").value;
// Search with reranking on 'Enter'
if (event.key === 'Enter') {
search(rerank=true);
}
// Limit incremental search to text types
else if (type !== "image") {
search(rerank=false);
}
}
function populate_type_dropdown() {
// Populate type dropdown field with enabled content types only
fetch("/api/config/types")
.then(response => response.json())
.then(enabled_types => {
// Show warning if no content types are enabled
if (enabled_types.detail) {
document.getElementById("results").innerHTML = "<div id='results-error'>To use Khoj search, setup your content plugins on the Khoj <a class='inline-chat-link' href='/config'>settings page</a>.</div>";
document.getElementById("query").setAttribute("disabled", "disabled");
document.getElementById("query").setAttribute("placeholder", "Configure Khoj to enable search");
return [];
}
document.getElementById("type").innerHTML =
enabled_types
.map(type => `<option value="${type}">${type.slice(0,1).toUpperCase() + type.slice(1)}</option>`)
.join('');
return enabled_types;
})
.then(enabled_types => {
// Set type field to content type passed in URL query parameter, if valid
var type_via_url = new URLSearchParams(window.location.search).get("t");
if (type_via_url && enabled_types.includes(type_via_url))
document.getElementById("type").value = type_via_url;
});
}
function createRequestUrl(query, type, results_count, rerank) {
// Generate Backend API URL to execute Search
let url = `/api/search?q=${encodeURIComponent(query)}&n=${results_count}&client=web`;
// If type is not 'all', append type to URL
if (type !== 'all')
url += `&t=${type}`;
// Rerank is only supported by text types
if (type !== "image")
url += `&r=${rerank}`;
return url;
}
function setTypeFieldInUrl(type) {
var url = new URL(window.location.href);
url.searchParams.set("t", type.value);
window.history.pushState({}, "", url.href);
}
function setQueryFieldInUrl(query) {
var url = new URL(window.location.href);
url.searchParams.set("q", query);
window.history.pushState({}, "", url.href);
}
window.onload = function () {
// Dynamically populate type dropdown based on enabled content types and type passed as URL query parameter
populate_type_dropdown();
// Fill query field with value passed in URL query parameters, if any.
var query_via_url = new URLSearchParams(window.location.search).get("q");
if (query_via_url)
document.getElementById("query").value = query_via_url;
}
</script>
<body>
<!--Add Header Logo and Nav Pane-->
<div class="khoj-header">
{% if demo %}
<!-- Banner linking to https://khoj.dev -->
<div class="khoj-banner-container">
<a class="khoj-banner" href="https://khoj.dev" target="_blank">
<p id="khoj-banner" class="khoj-banner">
Enroll in Khoj cloud to get your own Github assistant
</p>
</a>
<input type="text" id="khoj-banner-email" placeholder="email" class="khoj-banner-email"></input>
<button id="khoj-banner-submit" class="khoj-banner-button">Submit</button>
</div>
<a class="khoj-logo" href="https://khoj.dev" target="_blank">
<img class="khoj-logo" src="/static/assets/icons/khoj-logo-sideways.svg" alt="Khoj"></img>
</a>
{% else %}
<a class="khoj-logo" href="/">
<img class="khoj-logo" src="/static/assets/icons/khoj-logo-sideways.svg" alt="Khoj"></img>
</a>
{% endif %}
<nav class="khoj-nav">
<a class="khoj-nav" href="/chat">Chat</a>
<a class="khoj-nav khoj-nav-selected" href="/">Search</a>
{% if not demo %}
<a class="khoj-nav" href="/config">Settings</a>
{% endif %}
</nav>
</div>
<!--Add Text Box To Enter Query, Trigger Incremental Search OnChange -->
<input type="text" id="query" class="option" onkeyup=incrementalSearch(event) autofocus="autofocus" placeholder="What is the meaning of life?">
<div id="options">
<!--Add Dropdown to Select Query Type -->
<select id="type" class="option" onchange="setTypeFieldInUrl(this)"></select>
</div>
<!--Add Hints to Guide Search -->
<div id="info-hint" style="display: none">
Unexpected results? Hit Enter to get better results.
Else click Reinitialize on the <a href="/config">settings page</a> to fix it.
</div>
<!-- Section to Render Results -->
<div id="results"></div>
</body>
<style>
@media only screen and (max-width: 600px) {
body {
display: grid;
grid-template-columns: 1fr;
grid-template-rows: 1fr auto auto auto minmax(80px, 100%);
font-size: small!important;
}
body > * {
grid-column: 1;
}
}
@media only screen and (min-width: 600px) {
body {
display: grid;
grid-template-columns: 1fr min(70vw, 100%) 1fr;
grid-template-rows: 1fr auto auto auto minmax(80px, 100%);
padding-top: 60vw;
}
body > * {
grid-column: 2;
}
}
body {
padding: 0px;
margin: 0px;
background: #fff;
color: #475569;
font-family: roboto, karma, segoe ui, sans-serif;
font-size: 20px;
font-weight: 300;
line-height: 1.5em;
}
body > * {
padding: 10px;
margin: 10px;
}
#options {
padding: 0;
display: grid;
grid-template-columns: 1fr;
}
#options > * {
padding: 15px;
border-radius: 5px;
border: 1px solid #475569;
background: #f9fafc
}
.option:hover {
box-shadow: 0 0 11px #aaa;
}
#options > button {
margin-right: 10px;
}
#query {
font-size: larger;
}
#info-hint {
font-size: small;
color: #aaa;
text-align: center;
margin: 5px 0 0 0;
padding: 0;
}
#results {
font-size: medium;
margin: 0px;
line-height: 20px;
}
.results-image {
display: grid;
grid-template-columns: repeat(3, 1fr);
}
.image-link {
place-self: center;
}
.image {
width: 20vw;
border-radius: 10px;
border: 1px solid #475569;
}
#json {
white-space: pre-wrap;
}
.results-pdf,
.results-notion,
.results-plugin {
text-align: left;
white-space: pre-line;
}
.results-markdown,
.results-github {
text-align: left;
}
.results-org {
text-align: left;
white-space: pre-line;
}
.results-org h3 {
margin: 20px 0 0 0;
font-size: larger;
}
span.org-task-status {
color: white;
padding: 3.5px 3.5px 0;
margin-right: 5px;
border-radius: 5px;
background-color: #eab308;
font-size: medium;
}
span.org-task-status.todo {
background-color: #3b82f6
}
span.org-task-status.done {
background-color: #22c55e;
}
span.org-task-tag {
color: white;
padding: 3.5px 3.5px 0;
margin-right: 5px;
border-radius: 5px;
border: 1px solid #475569;
background-color: #ef4444;
font-size: small;
}
pre {
max-width: 100;
}
a {
color: #3b82f6;
text-decoration: none;
}
img.avatar {
width: 20px;
height: 20px;
border-radius: 50%;
}
div#results-error,
div.results-markdown,
div.results-notion,
div.results-org,
div.results-pdf {
text-align: left;
box-shadow: 2px 2px 2px var(--primary-hover);
border-radius: 5px;
padding: 10px;
margin: 10px 0;
border: 4px solid rgb(229, 229, 229);
}
div#results-error {
box-shadow: 2px 2px 2px #FF5722;
}
img {
max-width: 90%;
}
div.khoj-banner-container {
background: linear-gradient(-45deg, #FFC107, #FF9800, #FF5722, #FF9800, #FFC107);
background-size: 400% 400%;
animation: gradient 15s ease infinite;
text-align: center;
padding: 10px;
}
@keyframes gradient {
0% {
background-position: 0% 50%;
}
50% {
background-position: 100% 50%;
}
100% {
background-position: 0% 50%;
}
}
a.khoj-banner {
color: black;
}
a.khoj-logo {
text-align: center;
}
p.khoj-banner {
margin: 0;
padding: 10px;
}
button#khoj-banner-submit,
input#khoj-banner-email {
padding: 10px;
border-radius: 5px;
border: 1px solid #475569;
background: #f9fafc;
}
button#khoj-banner-submit:hover,
input#khoj-banner-email:hover {
box-shadow: 0 0 11px #aaa;
}
p#khoj-banner {
display: inline;
}
@media only screen and (max-width: 600px) {
a.khoj-banner {
display: block;
}
}
</style>
<script>
var khojBannerSubmit = document.getElementById("khoj-banner-submit");
khojBannerSubmit?.addEventListener("click", function(event) {
event.preventDefault();
var email = document.getElementById("khoj-banner-email").value;
fetch("https://lantern.khoj.dev/beta/users/", {
method: "POST",
body: JSON.stringify({
email: email
}),
headers: {
"Content-Type": "application/json"
}
}).then(function(response) {
return response.json();
}).then(function(data) {
console.log(data);
if (data.user != null) {
document.getElementById("khoj-banner").innerHTML = "Thanks for signing up. We'll be in touch soon! 🚀";
document.getElementById("khoj-banner-submit").remove();
} else {
document.getElementById("khoj-banner").innerHTML = "There was an error signing up. Please contact team@khoj.dev";
}
}).catch(function(error) {
console.log(error);
document.getElementById("khoj-banner").innerHTML = "There was an error signing up. Please contact team@khoj.dev";
});
});
</script>
</html>

6
src/interface/web/khoj.webmanifest → src/khoj/interface/web/khoj.webmanifest
View File

@@ -1,11 +1,11 @@
{
"name": "Khoj",
"short_name": "Khoj",
"description": "A natural language search engine for your personal notes, transactions and photos",
"description": "An AI search assistant for your digital brain",
"icons": [
{
"src": "/static/assets/icons/favicon-144x144.png",
"sizes": "144x144",
"src": "/static/assets/icons/favicon-128x128.png",
"sizes": "128x128",
"type": "image/png"
}
],

16
src/khoj/interface/web/khoj_chat.webmanifest Normal file
View File

@@ -0,0 +1,16 @@
{
"name": "Khoj Chat",
"short_name": "Khoj Chat",
"description": "An AI personal assistant for your digital brain",
"icons": [
{
"src": "/static/assets/icons/favicon-128x128.png",
"sizes": "128x128",
"type": "image/png"
}
],
"theme_color": "#ffffff",
"background_color": "#ffffff",
"display": "standalone",
"start_url": "/chat"
}

87
src/khoj/interface/web/processor_conversation_input.html Normal file
View File

@@ -0,0 +1,87 @@
{% extends "base_config.html" %}
{% block content %}
<div class="page">
<div class="section">
<h2 class="section-title">
<img class="card-icon" src="/static/assets/icons/chat.svg" alt="Chat">
<span class="card-title-text">Chat</span>
</h2>
<form id="config-form">
<table>
<tr>
<td>
<label for="openai-api-key" title="Get your OpenAI key from https://platform.openai.com/account/api-keys">OpenAI API key</label>
</td>
<td>
<input type="text" id="openai-api-key" name="openai-api-key" value="{{ current_config['openai_api_key'] }}">
</td>
</tr>
<tr>
<td>
<label for="chat-model">Chat Model</label>
</td>
<td>
<input type="text" id="chat-model" name="chat-model" value="{{ current_config['chat_model'] }}">
</td>
</tr>
</table>
<table style="display: none;">
<tr>
<td>
<label for="conversation-logfile">Conversation Logfile</label>
</td>
<td>
<input type="text" id="conversation-logfile" name="conversation-logfile" value="{{ current_config['conversation_logfile'] }}">
</td>
</tr>
<tr>
<td>
<label for="model">Model</label>
</td>
<td>
<input type="text" id="model" name="model" value="{{ current_config['model'] }}">
</td>
</tr>
</table>
<div class="section">
<div id="success" style="display: none;" ></div>
<button id="submit" type="submit">Save</button>
</div>
</form>
</div>
</div>
<script>
submit.addEventListener("click", function(event) {
event.preventDefault();
var openai_api_key = document.getElementById("openai-api-key").value;
var conversation_logfile = document.getElementById("conversation-logfile").value;
var model = document.getElementById("model").value;
var chat_model = document.getElementById("chat-model").value;
const csrfToken = document.cookie.split('; ').find(row => row.startsWith('csrftoken'))?.split('=')[1];
fetch('/api/config/data/processor/conversation', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
'X-CSRFToken': csrfToken
},
body: JSON.stringify({
"openai_api_key": openai_api_key,
"conversation_logfile": conversation_logfile,
"model": model,
"chat_model": chat_model
})
})
.then(response => response.json())
.then(data => {
if (data["status"] == "ok") {
document.getElementById("success").innerHTML = "✅ Successfully updated. Go to your <a href='/config'>settings page</a> to complete setup.";
document.getElementById("success").style.display = "block";
} else {
document.getElementById("success").innerHTML = "⚠️ Failed to update settings.";
document.getElementById("success").style.display = "block";
}
})
});
</script>
{% endblock %}

86
src/main.py → src/khoj/main.py
View File

@@ -6,44 +6,42 @@ import logging
import threading
import warnings
from platform import system
import webbrowser
# Ignore non-actionable warnings
warnings.filterwarnings("ignore", message=r'snapshot_download.py has been made private', category=FutureWarning)
warnings.filterwarnings("ignore", message=r'legacy way to download files from the HF hub,', category=FutureWarning)
warnings.filterwarnings("ignore", message=r"snapshot_download.py has been made private", category=FutureWarning)
warnings.filterwarnings("ignore", message=r"legacy way to download files from the HF hub,", category=FutureWarning)
# External Packages
import uvicorn
from fastapi import FastAPI
from fastapi.staticfiles import StaticFiles
from PyQt6 import QtWidgets
from PyQt6.QtCore import QThread, QTimer
from rich.logging import RichHandler
import schedule
# Internal Packages
from src.configure import configure_server
from src.routers.api import api
from src.routers.api_beta import api_beta
from src.routers.web_client import web_client
from src.utils import constants, state
from src.utils.cli import cli
from src.utils.helpers import CustomFormatter
from src.interface.desktop.main_window import MainWindow
from src.interface.desktop.system_tray import create_system_tray
from khoj.configure import configure_routes, configure_server
from khoj.utils import state
from khoj.utils.cli import cli
from khoj.interface.desktop.main_window import MainWindow
from khoj.interface.desktop.system_tray import create_system_tray
# Initialize the Application Server
app = FastAPI()
app.mount("/static", StaticFiles(directory=constants.web_directory), name="static")
app.include_router(api, prefix="/api")
app.include_router(api_beta, prefix="/api/beta")
app.include_router(web_client)
logger = logging.getLogger('src')
# Setup Logger
rich_handler = RichHandler(rich_tracebacks=True)
rich_handler.setFormatter(fmt=logging.Formatter(fmt="%(message)s", datefmt="[%X]"))
logging.basicConfig(handlers=[rich_handler])
logger = logging.getLogger("khoj")
def run():
# Turn Tokenizers Parallelism Off. App does not support it.
os.environ["TOKENIZERS_PARALLELISM"] = 'false'
os.environ["TOKENIZERS_PARALLELISM"] = "false"
# Load config from CLI
state.cli_args = sys.argv[1:]
@@ -53,52 +51,56 @@ def run():
# Create app directory, if it doesn't exist
state.config_file.parent.mkdir(parents=True, exist_ok=True)
# Setup Logger
# Set Logging Level
if args.verbose == 0:
logger.setLevel(logging.WARN)
elif args.verbose == 1:
logger.setLevel(logging.INFO)
elif args.verbose >= 2:
elif args.verbose >= 1:
logger.setLevel(logging.DEBUG)
# Set Log Format
ch = logging.StreamHandler()
ch.setFormatter(CustomFormatter())
logger.addHandler(ch)
# Set Log File
fh = logging.FileHandler(state.config_file.parent / 'khoj.log')
fh = logging.FileHandler(state.config_file.parent / "khoj.log", encoding="utf-8")
fh.setLevel(logging.DEBUG)
logger.addHandler(fh)
logger.info("Starting Khoj...")
logger.info("🌘 Starting Khoj")
if args.no_gui:
if not args.gui:
# Setup task scheduler
poll_task_scheduler()
# Start Server
configure_server(args, required=False)
configure_routes(app)
start_server(app, host=args.host, port=args.port, socket=args.socket)
else:
# Setup GUI
gui = QtWidgets.QApplication([])
main_window = MainWindow(args.config_file)
main_window = MainWindow(args.host, args.port)
# System tray is only available on Windows, MacOS.
# On Linux (Gnome) the System tray is not supported.
# Since only the Main Window is available
# Quitting it should quit the application
if system() in ['Windows', 'Darwin']:
if system() in ["Windows", "Darwin"]:
gui.setQuitOnLastWindowClosed(False)
tray = create_system_tray(gui, main_window)
tray.show()
# Setup Server
configure_server(args, required=False)
configure_routes(app)
server = ServerThread(app, args.host, args.port, args.socket)
url = f"http://{args.host}:{args.port}"
logger.info(f"🌗 Khoj is running at {url}")
try:
startup_url = url if args.config else f"{url}/config"
webbrowser.open(startup_url)
except:
logger.warning(f"🚧 Unable to open browser. Please open {url} manually to configure or use Khoj.")
# Show Main Window on First Run Experience or if on Linux
if args.config is None or system() not in ['Windows', 'Darwin']:
if args.config is None or system() not in ["Windows", "Darwin"]:
main_window.show()
# Setup Signal Handlers
@@ -113,9 +115,10 @@ def run():
gui.aboutToQuit.connect(server.terminate)
# Close Splash Screen if still open
if system() != 'Darwin':
if system() != "Darwin":
try:
import pyi_splash
# Update the text on the splash screen
pyi_splash.update_text("Khoj setup complete")
# Close Splash Screen
@@ -127,7 +130,6 @@ def run():
def sigint_handler(*args):
print("\nShutting down Khoj...")
QtWidgets.QApplication.quit()
@@ -137,13 +139,16 @@ def set_state(args):
state.verbose = args.verbose
state.host = args.host
state.port = args.port
state.demo = args.demo
def start_server(app, host=None, port=None, socket=None):
logger.info("🌖 Khoj is ready to use")
if socket:
uvicorn.run(app, proxy_headers=True, uds=socket)
uvicorn.run(app, proxy_headers=True, uds=socket, log_level="debug", use_colors=True, log_config=None)
else:
uvicorn.run(app, host=host, port=port)
uvicorn.run(app, host=host, port=port, log_level="debug", use_colors=True, log_config=None)
logger.info("🌒 Stopping Khoj")
def poll_task_scheduler():
@@ -168,5 +173,10 @@ class ServerThread(QThread):
start_server(self.app, self.host, self.port, self.socket)
if __name__ == '__main__':
def run_gui():
sys.argv += ["--gui"]
run()
if __name__ == "__main__":
run_gui()

0
src/processor/__init__.py → src/khoj/processor/__init__.py
View File

0
src/processor/conversation/__init__.py → src/khoj/processor/conversation/__init__.py
View File

145
src/khoj/processor/conversation/gpt.py Normal file
View File

@@ -0,0 +1,145 @@
# Standard Packages
import logging
from datetime import datetime
from typing import Optional
# External Packages
from langchain.schema import ChatMessage
# Internal Packages
from khoj.utils.constants import empty_escape_sequences
from khoj.processor.conversation import prompts
from khoj.processor.conversation.utils import (
chat_completion_with_backoff,
completion_with_backoff,
generate_chatml_messages_with_context,
)
logger = logging.getLogger(__name__)
def summarize(session, model, api_key=None, temperature=0.5, max_tokens=200):
"""
Summarize conversation session using the specified OpenAI chat model
"""
messages = [ChatMessage(content=prompts.summarize_chat.format(), role="system")] + session
# Get Response from GPT
logger.debug(f"Prompt for GPT: {messages}")
response = completion_with_backoff(
messages=messages,
model_name=model,
temperature=temperature,
max_tokens=max_tokens,
model_kwargs={"stop": ['"""'], "frequency_penalty": 0.2},
openai_api_key=api_key,
)
# Extract, Clean Message from GPT's Response
return str(response.content).replace("\n\n", "")
def extract_questions(
text, model: Optional[str] = "gpt-4", conversation_log={}, api_key=None, temperature=0, max_tokens=100
):
"""
Infer search queries to retrieve relevant notes to answer user query
"""
# Extract Past User Message and Inferred Questions from Conversation Log
chat_history = "".join(
[
f'Q: {chat["intent"]["query"]}\n\n{chat["intent"].get("inferred-queries") or list([chat["intent"]["query"]])}\n\n{chat["message"]}\n\n'
for chat in conversation_log.get("chat", [])[-4:]
if chat["by"] == "khoj"
]
)
# Get dates relative to today for prompt creation
today = datetime.today()
current_new_year = today.replace(month=1, day=1)
last_new_year = current_new_year.replace(year=today.year - 1)
prompt = prompts.extract_questions.format(
current_date=today.strftime("%A, %Y-%m-%d"),
last_new_year=last_new_year.strftime("%Y"),
last_new_year_date=last_new_year.strftime("%Y-%m-%d"),
current_new_year_date=current_new_year.strftime("%Y-%m-%d"),
bob_tom_age_difference={current_new_year.year - 1984 - 30},
bob_age={current_new_year.year - 1984},
chat_history=chat_history,
text=text,
)
messages = [ChatMessage(content=prompt, role="assistant")]
# Get Response from GPT
response = completion_with_backoff(
messages=messages,
model_name=model,
temperature=temperature,
max_tokens=max_tokens,
model_kwargs={"stop": ["A: ", "\n"]},
openai_api_key=api_key,
)
# Extract, Clean Message from GPT's Response
try:
questions = (
response.content.strip(empty_escape_sequences)
.replace("['", '["')
.replace("']", '"]')
.replace("', '", '", "')
.replace('["', "")
.replace('"]', "")
.split('", "')
)
except:
logger.warning(f"GPT returned invalid JSON. Falling back to using user message as search query.\n{response}")
questions = [text]
logger.debug(f"Extracted Questions by GPT: {questions}")
return questions
def converse(
references,
user_query,
conversation_log={},
model: str = "gpt-3.5-turbo",
api_key: Optional[str] = None,
temperature: float = 0.2,
completion_func=None,
):
"""
Converse with user using OpenAI's ChatGPT
"""
# Initialize Variables
current_date = datetime.now().strftime("%Y-%m-%d")
compiled_references = "\n\n".join({f"# {item}" for item in references})
# Get Conversation Primer appropriate to Conversation Type
if compiled_references == "":
conversation_primer = prompts.general_conversation.format(current_date=current_date, query=user_query)
else:
conversation_primer = prompts.notes_conversation.format(
current_date=current_date, query=user_query, references=compiled_references
)
# Setup Prompt with Primer or Conversation History
messages = generate_chatml_messages_with_context(
conversation_primer,
prompts.personality.format(),
conversation_log,
model,
)
truncated_messages = "\n".join({f"{message.content[:40]}..." for message in messages})
logger.debug(f"Conversation Context for GPT: {truncated_messages}")
# Get Response from GPT
return chat_completion_with_backoff(
messages=messages,
compiled_references=references,
model_name=model,
temperature=temperature,
openai_api_key=api_key,
completion_func=completion_func,
)

152
src/khoj/processor/conversation/prompts.py Normal file
View File

@@ -0,0 +1,152 @@
# External Packages
from langchain.prompts import PromptTemplate
## Personality
## --
personality = PromptTemplate.from_template("You are Khoj, a friendly, smart and helpful personal assistant.")
## General Conversation
## --
general_conversation = PromptTemplate.from_template(
"""
Using your general knowledge and our past conversations as context, answer the following question.
Current Date: {current_date}
Question: {query}
""".strip()
)
## Notes Conversation
## --
notes_conversation = PromptTemplate.from_template(
"""
Using the notes and our past conversations as context, answer the following question.
Current Date: {current_date}
Notes:
{references}
Question: {query}
""".strip()
)
## Summarize Chat
## --
summarize_chat = PromptTemplate.from_template(
f"{personality.format()} Summarize the conversation from your first person perspective"
)
## Summarize Notes
## --
summarize_notes = PromptTemplate.from_template(
"""
Summarize the below notes about {user_query}:
{text}
Summarize the notes in second person perspective:"""
)
## Answer
## --
answer = PromptTemplate.from_template(
"""
You are a friendly, helpful personal assistant.
Using the users notes below, answer their following question. If the answer is not contained within the notes, say "I don't know."
Notes:
{text}
Question: {user_query}
Answer (in second person):"""
)
## Extract Questions
## --
extract_questions = PromptTemplate.from_template(
"""
You are Khoj, an extremely smart and helpful search assistant with the ability to retrieve information from the user's notes.
- The user will provide their questions and answers to you for context.
- Add as much context from the previous questions and answers as required into your search queries.
- Break messages into multiple search queries when required to retrieve the relevant information.
- Add date filters to your search queries from questions and answers when required to retrieve the relevant information.
What searches, if any, will you need to perform to answer the users question?
Provide search queries as a JSON list of strings
Current Date: {current_date}
Q: How was my trip to Cambodia?
["How was my trip to Cambodia?"]
A: The trip was amazing. I went to the Angkor Wat temple and it was beautiful.
Q: Who did i visit that temple with?
["Who did I visit the Angkor Wat Temple in Cambodia with?"]
A: You visited the Angkor Wat Temple in Cambodia with Pablo, Namita and Xi.
Q: What national parks did I go to last year?
["National park I visited in {last_new_year} dt>='{last_new_year_date}' dt<'{current_new_year_date}'"]
A: You visited the Grand Canyon and Yellowstone National Park in {last_new_year}.
Q: How are you feeling today?
[]
A: I'm feeling a little bored. Helping you will hopefully make me feel better!
Q: How many tennis balls fit in the back of a 2002 Honda Civic?
["What is the size of a tennis ball?", "What is the trunk size of a 2002 Honda Civic?"]
A: 1085 tennis balls will fit in the trunk of a Honda Civic
Q: Is Bob older than Tom?
["When was Bob born?", "What is Tom's age?"]
A: Yes, Bob is older than Tom. As Bob was born on 1984-01-01 and Tom is 30 years old.
Q: What is their age difference?
["What is Bob's age?", "What is Tom's age?"]
A: Bob is {bob_tom_age_difference} years older than Tom. As Bob is {bob_age} years old and Tom is 30 years old.
{chat_history}
Q: {text}
"""
)
## Extract Search Type
## --
search_type = """
Objective: Extract search type from user query and return information as JSON
Allowed search types are listed below:
- search-type=["notes", "image", "pdf"]
Some examples are given below for reference:
Q:What fiction book was I reading last week about AI starship?
A:{ "search-type": "notes" }
Q: What did the lease say about early termination
A: { "search-type": "pdf" }
Q:Can you recommend a movie to watch from my notes?
A:{ "search-type": "notes" }
Q:When did I go surfing last?
A:{ "search-type": "notes" }
Q:"""

Some files were not shown because too many files have changed in this diff Show More