LLM/khoj
1
0
Fork 0
mirror of https://github.com/khoj-ai/khoj.git synced 2026-05-13 21:41:41 +00:00
Code Issues Packages Projects Releases Wiki Activity

Compare commits

base: LLM:1.26.2
Branches Tags
LLM:master LLM:release/1.x LLM:v2/add-live-voice-chat LLM:debanjum/miscellaneous-v2-changes LLM:use-user-specified-chat-model-for-intermediate-steps LLM:pr/hjamet/1109 LLM:create-data-tracer LLM:evals/tmp-distill-trains-of-thought LLM:load-web-app-from-desktop-app LLM:use-structured-index-to-improve-kb-retrieval LLM:speed-up-khoj-docker-builds LLM:features/support-sqlite-as-db LLM:regenerate-edit-messages-on-web-app LLM:improve-agents-pane-on-home-screen LLM:use-turnstyle-ux-for-khoj-chat LLM:improve-doc-chunking-and-support-timely-embedings LLM:speed-up-embedding-model-app-load-when-not-connected-to-internet LLM:panchayat
LLM:2.0.0-beta.28 LLM:2.0.0-beta.27 LLM:2.0.0-beta.26 LLM:2.0.0-beta.25 LLM:2.0.0-beta.24 LLM:2.0.0-beta.23 LLM:2.0.0-beta.22 LLM:2.0.0-beta.21 LLM:2.0.0-beta.20 LLM:2.0.0-beta.19 LLM:2.0.0-beta.18 LLM:2.0.0-beta.17 LLM:2.0.0-beta.16 LLM:2.0.0-beta.15 LLM:2.0.0-beta.14 LLM:2.0.0-beta.13 LLM:2.0.0-beta.12 LLM:2.0.0-beta.11 LLM:2.0.0-beta.10 LLM:2.0.0-beta.9 LLM:2.0.0-beta.8 LLM:1.42.10 LLM:2.0.0-beta.7 LLM:2.0.0-beta.6 LLM:2.0.0-beta.5 LLM:2.0.0-beta.4 LLM:2.0.0-beta.3 LLM:2.0.0-beta.2 LLM:2.0.0-beta.1 LLM:1.42.9 LLM:1.42.8 LLM:1.42.7 LLM:1.42.6 LLM:1.42.5 LLM:1.42.4 LLM:1.42.3 LLM:1.42.2 LLM:1.42.1 LLM:1.42.0 LLM:1.41.0 LLM:1.40.0 LLM:1.39.0 LLM:1.38.0 LLM:1.37.2 LLM:1.37.1 LLM:1.37.0 LLM:1.36.6 LLM:1.36.5 LLM:1.36.4 LLM:1.36.3 LLM:1.36.2 LLM:1.36.1 LLM:1.36.0 LLM:1.35.3 LLM:1.35.2 LLM:1.35.1 LLM:1.35.0 LLM:1.34.0 LLM:1.33.2 LLM:1.33.1 LLM:1.33.0 LLM:1.32.2 LLM:1.32.1 LLM:1.32.0 LLM:1.31.0 LLM:1.30.10 LLM:1.30.9 LLM:1.30.8 LLM:1.30.7 LLM:1.30.6 LLM:1.30.5 LLM:1.30.4 LLM:1.30.3 LLM:1.30.2 LLM:1.30.1 LLM:1.30.0 LLM:1.29.1 LLM:1.29.0 LLM:1.28.3 LLM:1.28.2 LLM:1.28.1 LLM:1.28.0 LLM:1.27.1 LLM:1.27.0 LLM:1.26.4 LLM:1.26.3 LLM:1.26.2 LLM:1.26.1 LLM:1.26.0 LLM:1.25.0 LLM:1.24.1 LLM:1.24.0 LLM:1.23.3 LLM:1.23.2 LLM:1.23.1 LLM:1.23.0 LLM:1.22.3 LLM:1.22.2 LLM:1.22.1 LLM:1.22.0 LLM:1.21.6 LLM:1.21.5 LLM:1.21.4 LLM:1.21.3 LLM:1.21.2 LLM:1.21.1 LLM:1.21.0 LLM:1.20.4 LLM:1.20.3 LLM:1.20.2 LLM:1.20.1 LLM:1.20.0 LLM:1.17.0 LLM:1.16.0 LLM:1.15.0 LLM:1.14.0 LLM:1.13.0 LLM:1.12.1 LLM:1.12.0 LLM:1.11.2 LLM:1.11.1 LLM:1.11.0 LLM:1.10.2 LLM:1.10.1 LLM:1.10.0 LLM:1.9.0 LLM:1.8.0 LLM:1.7.0 LLM:1.6.2 LLM:1.6.1 LLM:1.6.0 LLM:1.5.1 LLM:1.5.0 LLM:1.4.0 LLM:1.3.0 LLM:1.2.1 LLM:1.2.0 LLM:1.1.0 LLM:1.0.1 LLM:1.0.0 LLM:0.14.0 LLM:0.13.0 LLM:0.12.3 LLM:0.12.2 LLM:0.12.1 LLM:rm LLM:0.12.0 LLM:0.11.4 LLM:0.11.3 LLM:0.11.2 LLM:0.11.1 LLM:0.11.0 LLM:0.10.1 LLM:0.10.0 LLM:0.9.0 LLM:0.8.2 LLM:0.8.1 LLM:0.8.0 LLM:0.7.1 LLM:0.7.0 LLM:0.6.2 LLM:0.6.1 LLM:0.6.0 LLM:0.5.0 LLM:0.4.0 LLM:0.3.0 LLM:0.2.6 LLM:0.2.5 LLM:0.2.1 LLM:v0.2.0 LLM:v0.1.9 LLM:v0.1.8 LLM:v0.1.7 LLM:v0.1.6 LLM:v0.1.5-alpha
...
compare: LLM:1.33.1
Branches Tags
LLM:master LLM:release/1.x LLM:v2/add-live-voice-chat LLM:debanjum/miscellaneous-v2-changes LLM:use-user-specified-chat-model-for-intermediate-steps LLM:pr/hjamet/1109 LLM:create-data-tracer LLM:evals/tmp-distill-trains-of-thought LLM:load-web-app-from-desktop-app LLM:use-structured-index-to-improve-kb-retrieval LLM:speed-up-khoj-docker-builds LLM:features/support-sqlite-as-db LLM:regenerate-edit-messages-on-web-app LLM:improve-agents-pane-on-home-screen LLM:use-turnstyle-ux-for-khoj-chat LLM:improve-doc-chunking-and-support-timely-embedings LLM:speed-up-embedding-model-app-load-when-not-connected-to-internet LLM:panchayat
LLM:2.0.0-beta.28 LLM:2.0.0-beta.27 LLM:2.0.0-beta.26 LLM:2.0.0-beta.25 LLM:2.0.0-beta.24 LLM:2.0.0-beta.23 LLM:2.0.0-beta.22 LLM:2.0.0-beta.21 LLM:2.0.0-beta.20 LLM:2.0.0-beta.19 LLM:2.0.0-beta.18 LLM:2.0.0-beta.17 LLM:2.0.0-beta.16 LLM:2.0.0-beta.15 LLM:2.0.0-beta.14 LLM:2.0.0-beta.13 LLM:2.0.0-beta.12 LLM:2.0.0-beta.11 LLM:2.0.0-beta.10 LLM:2.0.0-beta.9 LLM:2.0.0-beta.8 LLM:1.42.10 LLM:2.0.0-beta.7 LLM:2.0.0-beta.6 LLM:2.0.0-beta.5 LLM:2.0.0-beta.4 LLM:2.0.0-beta.3 LLM:2.0.0-beta.2 LLM:2.0.0-beta.1 LLM:1.42.9 LLM:1.42.8 LLM:1.42.7 LLM:1.42.6 LLM:1.42.5 LLM:1.42.4 LLM:1.42.3 LLM:1.42.2 LLM:1.42.1 LLM:1.42.0 LLM:1.41.0 LLM:1.40.0 LLM:1.39.0 LLM:1.38.0 LLM:1.37.2 LLM:1.37.1 LLM:1.37.0 LLM:1.36.6 LLM:1.36.5 LLM:1.36.4 LLM:1.36.3 LLM:1.36.2 LLM:1.36.1 LLM:1.36.0 LLM:1.35.3 LLM:1.35.2 LLM:1.35.1 LLM:1.35.0 LLM:1.34.0 LLM:1.33.2 LLM:1.33.1 LLM:1.33.0 LLM:1.32.2 LLM:1.32.1 LLM:1.32.0 LLM:1.31.0 LLM:1.30.10 LLM:1.30.9 LLM:1.30.8 LLM:1.30.7 LLM:1.30.6 LLM:1.30.5 LLM:1.30.4 LLM:1.30.3 LLM:1.30.2 LLM:1.30.1 LLM:1.30.0 LLM:1.29.1 LLM:1.29.0 LLM:1.28.3 LLM:1.28.2 LLM:1.28.1 LLM:1.28.0 LLM:1.27.1 LLM:1.27.0 LLM:1.26.4 LLM:1.26.3 LLM:1.26.2 LLM:1.26.1 LLM:1.26.0 LLM:1.25.0 LLM:1.24.1 LLM:1.24.0 LLM:1.23.3 LLM:1.23.2 LLM:1.23.1 LLM:1.23.0 LLM:1.22.3 LLM:1.22.2 LLM:1.22.1 LLM:1.22.0 LLM:1.21.6 LLM:1.21.5 LLM:1.21.4 LLM:1.21.3 LLM:1.21.2 LLM:1.21.1 LLM:1.21.0 LLM:1.20.4 LLM:1.20.3 LLM:1.20.2 LLM:1.20.1 LLM:1.20.0 LLM:1.17.0 LLM:1.16.0 LLM:1.15.0 LLM:1.14.0 LLM:1.13.0 LLM:1.12.1 LLM:1.12.0 LLM:1.11.2 LLM:1.11.1 LLM:1.11.0 LLM:1.10.2 LLM:1.10.1 LLM:1.10.0 LLM:1.9.0 LLM:1.8.0 LLM:1.7.0 LLM:1.6.2 LLM:1.6.1 LLM:1.6.0 LLM:1.5.1 LLM:1.5.0 LLM:1.4.0 LLM:1.3.0 LLM:1.2.1 LLM:1.2.0 LLM:1.1.0 LLM:1.0.1 LLM:1.0.0 LLM:0.14.0 LLM:0.13.0 LLM:0.12.3 LLM:0.12.2 LLM:0.12.1 LLM:rm LLM:0.12.0 LLM:0.11.4 LLM:0.11.3 LLM:0.11.2 LLM:0.11.1 LLM:0.11.0 LLM:0.10.1 LLM:0.10.0 LLM:0.9.0 LLM:0.8.2 LLM:0.8.1 LLM:0.8.0 LLM:0.7.1 LLM:0.7.0 LLM:0.6.2 LLM:0.6.1 LLM:0.6.0 LLM:0.5.0 LLM:0.4.0 LLM:0.3.0 LLM:0.2.6 LLM:0.2.5 LLM:0.2.1 LLM:v0.2.0 LLM:v0.1.9 LLM:v0.1.8 LLM:v0.1.7 LLM:v0.1.6 LLM:v0.1.5-alpha

678 Commits
1.26.2 ... 1.33.1

Author SHA1 Message Date
sabaimran
25c1c1c591 Release Khoj version 1.33.1 2025-01-06 09:08:01 -08:00
sabaimran
689d9d8b3a Update formatting in admin.py and utils.py 2025-01-06 09:07:28 -08:00
thinker007
aa442c28eb Handle reporting chat estimated cost when some fields unavailable (#1026) 
Fix AttributeError: 'NoneType' object has no attribute 'model_extra'

* cost = chunk.usage.model_extra.get("estimated_cost", 0) if hasattr(chunk, "usage") and chunk.usage else 0  # Estimated costs returned by DeepInfra API
 2025-01-06 09:03:49 -08:00
sabaimran
4aed6f7e08 Add a link around the header khojlogotype to go home 2025-01-06 08:55:00 -08:00
Debanjum
266d274e21 Make automation should_notify check robust to non json mode chat models 
Use clean_json to handle automation should_notify check For gemini and
other chat models where enforcing json mode is problematic, not supported
 2025-01-06 20:16:32 +07:00
Debanjum
9a5e3583cf Remove bullet styling only from sidebar items on web app 
Previous fix had removed bullet styling from all components in web
app. This made chat messages on the web app lose bullet styling too.
 2025-01-06 20:15:42 +07:00
Debanjum
dc0bc5bcca Evaluate information retrieval quality using eval script 
- Encode article urls in filename indexed in Khoj KB
  Makes it easier for humans to compare, trace retrieval performance
  by looking at logs than using content hash (which was previously
  explored)
 2025-01-06 13:19:52 +07:00
Debanjum
daeba66c0d Optionally pass references used by agent for response to eval scorers 
This will allow the eval framework to evaluate retrieval quality too
 2025-01-06 13:19:52 +07:00
Debanjum
8231f4bb6e Return accuracy as decision to generalize across IR & standard scorers 2025-01-06 13:19:52 +07:00
sabaimran
eb9aadf72a Add an Obsidian README documentation for development 2025-01-05 19:06:27 -08:00
sabaimran
e89e49818b Merge pull request #1028 from ReallyVirtual/patch-1 
Update image_generation.md
 2025-01-05 13:53:00 -08:00
sabaimran
616cc189d1 Remove bullet points from li styling explicitly 2025-01-05 13:52:08 -08:00
sabaimran
a5705a5aa6 After agent prompt safe check is parsed as json, load it into a json object 2025-01-05 13:40:20 -08:00
Sohaib Athar
95a2387e9b Update image_generation.md 
Fixed typo
 2025-01-05 07:25:41 +05:00
sabaimran
756f4a2a66 Update regeneration logic to run for all entries now that we have a single search model ID 2025-01-03 15:16:10 -08:00
Debanjum
33f85b4a55 Fix title of Query Filters documentation 2025-01-01 18:56:16 +08:00
sabaimran
0ef787b57c Set waitBeforeSeconds to 0 2024-12-30 15:08:34 -08:00
Debanjum
90b4e03454 Pull out query filters as top level documentation page 
- Note perf eval from 2022
- Update links to query-filters in docs
- Fix links
- Update image model docs
 2024-12-30 14:35:05 -08:00
sabaimran
8f69eb949b Release Khoj version 1.33.0 2024-12-29 10:39:26 -08:00
Debanjum
1a43ca75f3 Update to latest jinja python package dependency 2024-12-27 01:44:41 -08:00
Debanjum
ca197ba9ba Improve Automation Flexibility and Automation Email Format (#1016) 
- Format AI response to send in automation email
- Let Khoj infer chat query based on user automation query
- Decide if automation emails should be sent based on response
  - Fix the `to_notify_or_not` decider AI
  - Ask reason before decision to improve to_notify decider AI
- Show error message on web app when fail to create/update automation
 2024-12-27 01:36:38 -08:00
Debanjum
90685ccbb0 Show error message if update automation via web app fails 2024-12-26 22:21:56 -08:00
Debanjum
c4bb92076e Convert async create automation api endpoints to sync 2024-12-26 21:59:55 -08:00
Debanjum
9674de400c Format AI response to send in automation email 
Previously we sent the AI response directly. This change post
processes the AI response with the awareness that it is to be sent to
the user as an email to improve rendering and quality of the emails.
 2024-12-26 21:04:50 -08:00
Debanjum
6d219dcc1d Switch to let Khoj infer chat query based on user automation query 
This tries to decouple the automation query from the chat query. So
the chat model doesn't have to know it is running in an automation
context or figure how to notify user or send automation response. It
just has to respond to the AI generated `query_to_run' corresponding
to the `scheduling_request` automation by the user.

For example, a `scheduling_request' of `notify me when X happens'
results in the automation calling the chat api with a `query_to_run`
like `tell me about X` and deciding if to notify based on information
gathered about X from the scheduled run. If these two are not
decoupled, the chat model may respond with how it can notify about X
instead of just asking about it.

Swap query_to_run with scheduling_request on the automation web page
 2024-12-26 21:04:50 -08:00
Debanjum
3600a9a4f3 Ask reason before decision to improve to_notify decider automation AI 
Previously it just gave a decision. This was hard to debug during
prompt tuning. Asking for reason before decision improves models
decision quality.
 2024-12-26 21:04:50 -08:00
Debanjum
dcc5073d16 Fix the to_notify decider automation chat actor. Add detailed logging 2024-12-26 21:04:50 -08:00
sabaimran
03b4667acb Merge pull request #1017 from khoj-ai/features/update-home-page 
- Rather than chunky generic cards, make the suggested actions more action oriented, around the problem a user might want to solve. Give them follow-up options. Design still in progress.
 2024-12-24 12:12:37 -08:00
sabaimran
5985ef4c7c Further improve prompts 2024-12-24 12:11:27 -08:00
sabaimran
012e0ef882 Add tooltip for file input ref. 2024-12-24 11:20:39 -08:00
sabaimran
a58b3b4a37 Remove some of the step one suggestions 2024-12-24 10:52:42 -08:00
sabaimran
cf78f426d3 Merge branch 'master' of github.com:khoj-ai/khoj into features/update-home-page 2024-12-24 09:52:06 -08:00
sabaimran
d27d291584 Merge pull request #1015 from khoj-ai/features/clean-up-authenticated-data 
- De facto, was being assumed everywhere if authenticatedData is null, that it's not logged in. This isn't true because the data can still be loading. Update the hook to send additional states.
- Bonus: Delete model picker code and a slew of unused imports.
 2024-12-24 09:51:39 -08:00
sabaimran
cd4cf4f9f6 Merge pull request #1014 from khoj-ai/features/improve-agent-management 
- Add support for seeing all steps of the agent modification flow via tabs at the top of the modal
- Separate knowledge base & tool selection into two separate parts
 2024-12-24 09:50:38 -08:00
sabaimran
3e6ba45cbe Merge pull request #1013 from khoj-ai/features/use-sidebar 
Use the [shadcn sidebar](https://ui.shadcn.com/docs/components/sidebar#sidebarmenusub) across Khoj and standardize how the side panel experience works across the app. This helps us generalize the code better, while re-using the same components. 

Deprecates current usage of the chat history side panel, replacing it with the new `appSidebar.tsx` component.

We'll eventually move out the `Manage Files` section and move it into a separate panel dedicated to chat-level controls.
 2024-12-24 09:50:09 -08:00
sabaimran
95fdbe13ae move all conversations button to bottom of side panel 2024-12-23 23:13:19 -08:00
sabaimran
d0256f267e Fix submit state for form with buttons sticky to button 2024-12-23 23:10:19 -08:00
Debanjum
0d5fc70aa3 Fix colors, title on create agent card 2024-12-23 20:01:56 -08:00
Debanjum
0b91383deb Make post oauth next url redirect more robust, handle q params better 2024-12-23 17:50:45 -08:00
Debanjum
17f8ba732d Autofocus on email input box when email sign-in selected on web app 2024-12-23 17:49:41 -08:00
sabaimran
fb111a944b Use disable_https flag instead of is_in_debug_mode to decide whether to redirect google auth request 2024-12-23 17:23:47 -08:00
sabaimran
3fd0c202ea Allow better spacing in the agent card and make the buttons sticky 2024-12-23 16:43:50 -08:00
sabaimran
c83709fdd1 Further clean up in home page initial cards experience 2024-12-22 18:11:19 -08:00
sabaimran
4c4f4401b1 Make suggestion cards a little more minimal 2024-12-22 11:54:27 -08:00
sabaimran
90b02b4cfe Merge branch 'features/clean-up-authenticated-data' of github.com:khoj-ai/khoj into features/update-home-page 2024-12-22 11:26:29 -08:00
sabaimran
798837378f Improve mobile friendliness and highlight missing necessary data 2024-12-22 11:02:50 -08:00
sabaimran
7032ccf130 Show create agent button when not logged in agents page 2024-12-22 10:01:15 -08:00
sabaimran
0fefbac89f Improve sidebar experience for not logged in state 2024-12-22 09:58:21 -08:00
sabaimran
60f80548a4 Remove unused span text 2024-12-22 09:21:40 -08:00
sabaimran
9f84f5dcc7 Give more breathing space to the sidebar footer 2024-12-22 09:18:05 -08:00
sabaimran
dc17272f70 Fix some spacing in the nav menu 2024-12-22 09:01:09 -08:00
sabaimran
45da6ec750 Separate the shorthand of each suggestion card from the prefilled text 2024-12-21 20:46:57 -08:00
sabaimran
bf405f50d7 Fix spacing of main content in the settings page 2024-12-21 20:46:21 -08:00
sabaimran
62dd4c55d4 Further improve UX of the suggestion cards 2024-12-21 20:10:54 -08:00
sabaimran
8c9c57e060 Merge branch 'features/clean-up-authenticated-data' of github.com:khoj-ai/khoj into features/update-home-page 2024-12-21 19:32:07 -08:00
sabaimran
8c6b4217ae Set width of chat layout to 100% 2024-12-21 19:29:37 -08:00
sabaimran
a17cc9db38 Fix handling 403 forbidden error from auth response 2024-12-21 19:19:35 -08:00
sabaimran
95826393e1 Update the home page suggestion cards 
- Rather than chunky generic cards, make the suggested actions more action oriented, around the problem a user might want to solve. Give them follow-up options. Design still in progress.
 2024-12-21 18:57:19 -08:00
Debanjum
8d129c4675 Bump default max prompt size for commercial chat models 2024-12-21 17:31:05 -08:00
Debanjum
37ae48d9cf Add support for OpenAI o1 model 
Needed to handle the current API limitations of the o1 model.
Specifically its inability to stream responses
 2024-12-21 17:25:32 -08:00
sabaimran
2c7c16d93e Fix conditional reference to is mobile width hook 2024-12-21 12:48:29 -08:00
sabaimran
e9dae4240e Clean up all references to authenticatedData 
- De facto, was being assumed everywhere if authenticatedData is null, that it's not logged in. This isn't true because the data can still be loading. Update the hook to send additional states.
- Bonus: Delete model picker code and a slew of unused imports.
 2024-12-21 08:45:43 -08:00
sabaimran
b1c5c5bcc9 Fix path to component library in shadcn sidebar 2024-12-20 15:48:57 -08:00
sabaimran
cc7fd1163f Use tabs to separate sections in the agent mod form 
- Add knowledge base as a separate section, apart from tools
- This makes it easier to navigate the different components quickly
 2024-12-20 14:41:15 -08:00
sabaimran
078753df30 Add chatwoot to the frame-src CSP 2024-12-20 13:33:48 -08:00
sabaimran
efe812e323 Add componets for tabs in agent page 2024-12-20 13:33:20 -08:00
sabaimran
ba792c02ba Improve share chat UI for alignment 2024-12-20 12:28:31 -08:00
sabaimran
7770caa793 Add side bar inset to home page. Simplify automations card. 2024-12-20 11:37:23 -08:00
sabaimran
b8a9dcd600 Improve mobile layout with sidebar inset and fix dark mode logo 2024-12-19 23:23:52 -08:00
sabaimran
b1880d9c9d Add side bar to search page 2024-12-19 22:46:50 -08:00
sabaimran
02c503e966 Further improve mobile layout with side panel 2024-12-19 22:00:32 -08:00
sabaimran
43331f7730 Remove unused css classes 2024-12-19 21:36:35 -08:00
sabaimran
b644bb8628 Further improve mobile friendliness 2024-12-19 21:34:04 -08:00
sabaimran
9d5480d886 Improve mobile friendlinses across chat and home pages. 2024-12-19 20:33:53 -08:00
sabaimran
68af10c805 Use the new shadcn sidebar for khoj nav and actions 
- Use the sidebar across all pages to quickly navigate through the app, access settings, and go to past chats
- Pending: mobile friendliness
 2024-12-19 20:10:03 -08:00
sabaimran
7eb15bf0a9 Update shadcn components 2024-12-19 14:33:36 -08:00
sabaimran
a4aeb9fdf3 Simplify the home page color scheme and overall design 2024-12-19 14:02:53 -08:00
Debanjum
0ae21e5628 Use icons, not text labels, for sidebar nav items on docs website 2024-12-17 20:49:09 -08:00
sabaimran
cafe1b0655 Remove error log line for payload inclusion 2024-12-17 19:48:22 -08:00
Debanjum
1e3f452d15 Handle sharing old conversation publically even if they have no slug 
New conversation have a slug, but older conversation may not. This
change allows those older conversations to still be shareable by using
a random uuid for constructing their url instead
 2024-12-17 19:44:36 -08:00
Debanjum
90b7ba51a4 Make agent prompt safety checker more forgiving and concise 
- Some of the instructions were duplicated (e.g illegal, harmful)
- Return format requested was inconsistent
- Safety prompt felt overtly prudish which lowered their utility.
  Make it laxer for now, add checks later if required
 2024-12-17 19:44:36 -08:00
sabaimran
bcee2ea01a Release Khoj version 1.32.2 2024-12-17 16:03:29 -08:00
sabaimran
92144c8102 Remove release step in todesktop flow, since we need to run releases manually now 
- Leaving it commented out for the time being so we can revisit automating this later
 2024-12-17 16:02:45 -08:00
sabaimran
f291884921 If not in debug mode, force google auth to use the https protocol 2024-12-17 15:44:18 -08:00
sabaimran
bcc1bc6854 Log the payload sent temporarily in order to help with debugging 2024-12-17 15:37:42 -08:00
sabaimran
7ca2553d17 Update login popup copy 2024-12-17 15:20:44 -08:00
sabaimran
ef99d8c28e Add more descriptive error logs when google auth token verification fails 2024-12-17 15:18:38 -08:00
Debanjum
60d55a83c4 Use khoj logo via url in readme to load in other locations like pypi 2024-12-17 14:00:53 -08:00
Debanjum
10bd56d2b9 Attest Khoj pypi package by upgrading pypi publish gh action 
- Print hash in CI to ease verifying ci built python package matches
  khoj package published on pypi
- Newer pypi publish github action should speed up workflow by ~30s
 2024-12-17 13:40:39 -08:00
sabaimran
2e80a1ce7c Release Khoj version 1.32.1 2024-12-17 13:28:00 -08:00
Debanjum
df15f00243 Tag docker images with latest tag in dockerize workflow on release 2024-12-17 13:18:51 -08:00
sabaimran
ded168dae3 Release Khoj version 1.32.0 2024-12-17 12:29:20 -08:00
sabaimran
f6abfcfa6b Use latest release version for pypi gh action to publish 2024-12-17 12:19:42 -08:00
Debanjum
c20364efcb Upgrade web app next.js, shadcn and other package dependencies 2024-12-17 11:15:22 -08:00
Debanjum
63d2c6f35a Allow research mode and other conversation commands in automations (#1011) 
Major
---
Previously we couldn't enable research mode or use other slash
commands in automated tasks.

This change separates determining if a chat query is triggered via
automated task from the (other) conversation commands to run the query
with.

This unlocks the ability to enable research mode in automations
apart from other variations like /image or /diagram etc.

Minor
---
- Clean the code to get data sources and output formats
- Have some examples shown on automations page run in research mode now
 2024-12-17 00:44:51 -08:00
sabaimran
3b050a33bb Include resend as a default dependency, rather than restricting to prod 2024-12-16 22:24:41 -08:00
sabaimran
741e9f56f9 Update admin button for getting login url to include user email 2024-12-16 22:16:05 -08:00
Debanjum
88aa8c377a Support online search with Searxng as zero config, self-hostable solution (#1010) 
This allows online search to work out of the box again 
for self-hosting users, as no auth/api key setup required.

Docker users do not need to change anything in their setup flow.
Direct installers can setup Searxng locally or use public instances if
they do not want to use any of the other providers (like Jina, Serper)

Resolves #749. Resolves #990
 2024-12-16 18:59:09 -08:00
sabaimran
7677bb14f8 Update docs home page 2024-12-16 18:48:42 -08:00
sabaimran
1bd0d46b3d Update the theme color used in our docs to match the theme in our emails 2024-12-16 18:42:50 -08:00
sabaimran
432e901087 Update the online search self-hosting instructions to reflect new setup 2024-12-16 18:37:20 -08:00
sabaimran
af2553a890 Remove the Searxng API key env variable for simplicity 2024-12-16 18:36:55 -08:00
sabaimran
19d80d190e Add latest tag to the khoj cloud description for prod 2024-12-16 17:56:12 -08:00
sabaimran
d62cc0d539 Merge branch 'master' of github.com:khoj-ai/khoj into support-online-search-via-searxng 2024-12-16 17:55:06 -08:00
sabaimran
6f3218f487 Merge pull request #1008 from khoj-ai/features/new-sign-in-page 
- Make it easier to quickly create the account without losing track of where you are
- Show some capabilities before you sign on
 2024-12-16 17:54:43 -08:00
sabaimran
753859fbe0 Make the docs and github buttons on the sign in email less prominent 2024-12-16 17:49:13 -08:00
sabaimran
20888d3930 Clarify some of the language in the sign in email 2024-12-16 17:47:12 -08:00
sabaimran
13e7455f56 Use lowercase c in click 2024-12-16 17:44:08 -08:00
sabaimran
d17a9ba4c8 Fix return data for expired code user 2024-12-16 17:33:53 -08:00
sabaimran
efb0b9f495 Gracefully handle error when user login code is expired 2024-12-16 16:47:54 -08:00
sabaimran
064f7e48ca Update various copy texts for OG metadata and such 2024-12-16 16:40:46 -08:00
sabaimran
28b8f9105d Remove parenthetical from email template 2024-12-16 13:40:41 -08:00
Debanjum
9d02978f6e Support online search with Searxng as turnkey, self-hostable solution 
This allows online search to work out of the box again for
self-hosting users, as no auth/api key setup required.

Docker users do not need to change anything in their setup flow.
Direct installers can setup searxng locally or use public instances if
they do not want to use any of the other providers (like Jina, Serper)

Resolves #749. Resolves #990
 2024-12-16 12:53:38 -08:00
Debanjum
9c64275dec Auto redirect requests to use HTTPS if server is using SSL certs 2024-12-16 12:53:38 -08:00
sabaimran
ae9750e58e Add rate limiting to OTP login attempts and update email template for conciseness 2024-12-16 11:59:45 -08:00
sabaimran
b7783357fa Decrease timeout limit for verification codes to 5 minutes 2024-12-16 09:07:34 -08:00
sabaimran
5d3da3340f Include email in verification API 2024-12-15 13:54:41 -08:00
sabaimran
8e3313156e Simplify the magic link email a little bit 2024-12-14 11:19:31 -08:00
sabaimran
6a56140360 Allow users to directly enter their unique code when logging in 
- Code automatically becomes invalid after 30 minutes
 2024-12-14 11:06:05 -08:00
sabaimran
c25174e8d4 Apply more finished styling to login features, make the pop-up mobile friendly 2024-12-14 09:46:19 -08:00
sabaimran
73c1fe6ae1 Add text overlay to caption the different assets 2024-12-13 13:45:31 -08:00
Debanjum
132f2c987a Make Khoj email sender configurable for all email variants 
The welcome, feedback and automation emails were still using the Khoj
domain, which wouldnt work for self-hosted users with their RESEND key

Resolves #1004
 2024-12-13 12:25:23 -08:00
sabaimran
f1fb4525c6 Remove old images 2024-12-13 11:31:14 -08:00
sabaimran
d5681ad1a2 Update image assets to sign up prompt 2024-12-13 11:30:14 -08:00
sabaimran
62545a9af3 Update package path for pypi ci export 2024-12-12 21:25:21 -08:00
sabaimran
e74e922cea Update file path of python installation 2024-12-12 16:50:32 -08:00
sabaimran
144f283a25 Maintain old login page for posterity and associated API 2024-12-12 16:23:44 -08:00
sabaimran
4697daeb1a Improve opengraph metadata across front-end pages 2024-12-12 15:56:43 -08:00
sabaimran
dfc150c442 Merge branch 'master' of github.com:khoj-ai/khoj into features/new-sign-in-page 2024-12-12 15:43:06 -08:00
sabaimran
ad3f8a33d1 Add a static login footer that prompts login, disable input box without auth 2024-12-12 14:57:52 -08:00
Debanjum
2db7a1ca6b Restart code sandbox on crash in eval github workflow (#1007) 
See
e3fed3750b
for corresponding change to use pm2 to auto-restart code sandbox
 2024-12-12 14:32:03 -08:00
Debanjum
12c976dcb2 Track usage costs from DeepInfra OpenAI compatible API 2024-12-12 14:20:34 -08:00
Debanjum
b0abec39d5 Use chat model name var name consistently and type openai chat utils 2024-12-12 14:20:34 -08:00
Debanjum
4915be0301 Fix initializing chat model names parameter after field rename in #1003 2024-12-12 14:20:33 -08:00
sabaimran
a7d0ed8670 Add carousel for navigating images in the sign up modal 2024-12-12 11:47:41 -08:00
Debanjum
9eb863e964 Restart code sandbox on crash in eval github workflow 2024-12-12 11:28:54 -08:00
Debanjum
01bc6d35dc Rename Chat Model Options table to Chat Model as short & readable (#1003) 
- Previous was incorrectly plural but was defining only a single model
- Rename chat model table field to name
- Update documentation
- Update references functions and variables to match new name
 2024-12-12 11:24:16 -08:00
sabaimran
943065b7b3 Remove dead dependencies and improve the google sign in button 2024-12-12 11:19:19 -08:00
sabaimran
41bb1e60d0 Use the LoginPrompt in the chat history side panel 2024-12-11 22:56:03 -08:00
sabaimran
b60b750555 Update the styling to align with Google branding via the sign in button 
- Disable the gsi client side code since it's being finnicky and inconsistent
 2024-12-11 22:49:11 -08:00
sabaimran
0f8b055b42 Improve padding for space, esp in mobile 2024-12-11 18:22:48 -08:00
sabaimran
142239d2c9 Add mobile friendliness and replace the login page redirects 2024-12-11 18:01:04 -08:00
sabaimran
de6ed2352a Break up the parts of the login dialog into smaller modules to extend for mobile 2024-12-11 17:18:43 -08:00
sabaimran
d35db99c6f Initial version of a carousel working for sign in with steps for email sign up 
- Google sign in is pending with the gsi client code. Will see if I can get that working
- Check in relevant image assets
 2024-12-11 16:54:06 -08:00
aditya218
9be26e1bd2 Fix documentation to point to local environment image_generation.md (#1005) 
Fix documentation to point to local environment.
 2024-12-11 16:12:25 -08:00
sabaimran
530b44cf56 Merge branch 'master' of github.com:khoj-ai/khoj into features/new-sign-in-page 2024-12-11 10:30:13 -08:00
Debanjum
fe09df66b7 Make code sandbox container url accessible to Khoj container in docker compose 2024-12-11 01:14:26 -08:00
Debanjum
59008ae90e Use buildx to create multi platform docker image 2024-12-11 00:21:29 -08:00
Debanjum
ec797bc6b8 Build docker imgs on native arch runners to avoid manifest list error 
This also avoids the need to use --amend and annotate steps when
creating the multi-arch docker images
 2024-12-10 23:16:36 -08:00
Debanjum
5f7b13df2d Fix new docker tags in workflow to not include forward slashes 2024-12-10 22:55:33 -08:00
Debanjum
ba6237b5c0 Fix to create multi-arch builds. Stop docker image overwrites in workflow 2024-12-10 21:08:17 -08:00
sabaimran
44ede26e67 Temporarily disable cloud arm builds while we disambiguate the build issues 2024-12-10 20:00:59 -08:00
Debanjum
33a5efaf4b Fix undefined variable exception during openai provider setup on init 
Resolves #1001
 2024-12-10 19:54:00 -08:00
sabaimran
e43341fdcc Release Khoj version 1.31.0 2024-12-10 19:41:31 -08:00
Debanjum
a757ecfd2b Put the generated assets message after the user query and fix prompt 2024-12-10 19:40:13 -08:00
Debanjum
40e4f2ec2e Reduce clutter in chat message ux on Obsidian 
- Move khoj message border to left like in web ui
- Remove user, sender emojis and name
- Ensure conversation title always at top of chat sessions view,
  even if no chat sessions loaded yet, instead of causing layout shift
  on chat sessions load
 2024-12-10 19:34:17 -08:00
sabaimran
1ec1eff57e Improve mobile header, reduce title bar padding and add conv title 2024-12-10 19:03:57 -08:00
sabaimran
321eeeaed7 Fix setting title of shared conversation, move shared button into the title pane 2024-12-10 18:19:46 -08:00
sabaimran
d7e5a76ace Add an icon in the input bar for research mode 2024-12-10 17:49:25 -08:00
sabaimran
01d000e570 Merge pull request #1002 from khoj-ai/features/improve-multiple-output-mode-generation 
Improve handling of multiple output modes

- Use the generated descriptions / inferred queries to supply context to the model about what it's created and give a richer response
- Stop sending the generated image in user message. This seemed to be confusing the model more than helping.
- Collect generated assets in a structured objects to provide model context. This seems to help it follow instructions and separate responsibility better
- Also, rename the open ai converse method to converse_openai to follow patterns with other providers
 2024-12-10 17:06:19 -08:00
sabaimran
2bb14c55a6 Merge branch 'master' of github.com:khoj-ai/khoj into features/improve-multiple-output-mode-generation 2024-12-10 16:56:36 -08:00
sabaimran
6c8007e23b Improve handling of multiple output modes 
- Use the generated descriptions / inferred queries to supply context to the model about what it's created and give a richer response
- Stop sending the generated image in user message. This seemed to be confusing the model more than helping.
- Also, rename the open ai converse method to converse_openai to follow patterns with other providers
 2024-12-10 16:54:21 -08:00
Debanjum
4bc5c1357a Upgrade server, documentation dependencies. Spell fix docker-compose.yml 2024-12-10 15:47:47 -08:00
Debanjum
f8957e52bf Keep chatml message content simple for wider compat unless attachments 
This allows for wider compatibility with chat models and openai proxy
ai model apis that expect message content to be string format, not
objects.
 2024-12-10 00:10:56 -08:00
sabaimran
4b4e0e20d4 Make the version number a badge, rather than an independent item in the nav dropdown 2024-12-09 14:45:26 -08:00
sabaimran
eb36492ba5 Update handling of images when included in the chat history with assistant message 2024-12-08 21:46:07 -08:00
Debanjum
b660c494bc Use recognizable DB model names to ease selection UX on Admin Panel 
Previously id were used (by default) for model display strings.
This made it hard to select chat model options, server chat settings
etc. in the admin panel dropdowns.

This change uses more recognizable names for the DB objects to ease
selection in dropdowns and display in general on the admin panel.
 2024-12-08 20:34:50 -08:00
Debanjum
d10dc9cfe1 Inform code tool AI only limited python packages are available to it 
Reduce code tool failing with module not found errors
 2024-12-08 20:34:50 -08:00
Debanjum
3fd8614a4b Only auto load available chat models from Ollama provider for now 
Allowing models from any openai proxy service makes it too unwieldy.
And a bunch of them do not even support this endpoint.
 2024-12-08 20:34:50 -08:00
sabaimran
2c934162d3 Add a data filter for privacy_level of agents 2024-12-08 19:55:57 -08:00
sabaimran
3b9f4c4356 Correct negative for running prod image locally 2024-12-08 19:55:35 -08:00
Debanjum
9dd3782f5c Rename OpenAIProcessorConversationConfig DB model to more apt AiModelApi (#998) 
* Rename OpenAIProcessorConversationConfig to more apt AiModelAPI

The DB model name had drifted from what it is being used for,
a general chat api provider that supports other chat api providers like
anthropic and google chat models apart from openai based chat models.

This change renames the DB model and updates the docs to remove this
confusion.

Using Ai Model Api we catch most use-cases including chat, stt, image generation etc.
 2024-12-08 18:02:29 -08:00
sabaimran
df66fb23ab Centralize definition of the content security policy and add in-app chat 
- in-app chat is meant for support requests and currently is only in the settings page, where users are most likely to go if confused IMO
 2024-12-08 17:57:27 -08:00
sabaimran
0b87c13f8d Add khoj_version to the settings menu 2024-12-08 17:55:56 -08:00
sabaimran
47a087c73b Fix chatwoot import issue by checking whether we're in an execution environment before loading the script 2024-12-08 17:16:20 -08:00
sabaimran
66f59c8d41 Add Chatwoot to documentation 
See repo: https://github.com/chatwoot/chatwoot
 2024-12-08 16:51:43 -08:00
sabaimran
6ed051d631 Merge pull request #994 from khoj-ai/features/update-desktop-app 
Simplify the desktop app

- Make the desktop app mainly a file-syncing client for users who have lots of documents that they need to share with Khoj. This is because the web app provides a fairly robust chat client which can be used by anyone on their computer.
- The chat client in the desktop app had significantly drifted from our current brand / them, and didn't provide enough value add to update. Later, we will make it easier to install the existing web app as a desktop PWA.
 2024-12-08 15:05:35 -08:00
sabaimran
05b3911080 Update some button titles and add descriptions for clarity 2024-12-08 14:29:12 -08:00
sabaimran
b78b92d6a0 Merge branch 'master' of github.com:khoj-ai/khoj into features/update-desktop-app 2024-12-08 14:20:20 -08:00
sabaimran
e3789aef49 Merge pull request #992 from khoj-ai/features/allow-multi-outputs-in-chat 
Currently, Khoj has terminal states with respect to what assets it outputs. We limit it to image, text, and excalidraw diagram. This limitation is unnecessary and provides undue constraints for creating more dynamic, diverse experiences. For instance, we may want the chat view to morph for document editing or generation, in which case having limited output modes would be a detriment.

This change allows us to emit generated assets and then continue on to more text generation in final response. It forces text response for all messages. It adds a new stream event, GENERATED_ASSETS, which holds content that the AI is emitting in response to the query.
 2024-12-08 14:19:05 -08:00
sabaimran
a2251f01eb Make result optional for code context, relevant when code execution was unsuccessful 2024-12-08 13:27:33 -08:00
sabaimran
9c403d24e1 Fix reference to directory in the eval workflow for starting terrarium 2024-12-08 13:03:05 -08:00
sabaimran
7cd2855146 Make attributes optional in the knowledge graph model 2024-12-08 12:23:17 -08:00
sabaimran
2af687d1c5 Allow snippetHighlighted to also be nullable 2024-12-08 11:51:24 -08:00
sabaimran
efa23a8ad8 Update validation requirements for online searches 2024-12-08 11:30:17 -08:00
sabaimran
6940c6379b Add sudo when running installations in order to install relevant packages 
add --legacy-peer-deps temporarily to see if it helps mitigate the issue
 2024-12-08 11:11:13 -08:00
sabaimran
4c4b7120c6 Use Khoj terrarium fork instead of building from official Cohere repo 2024-12-08 11:06:33 -08:00
sabaimran
a138845fea Merge branch 'master' of github.com:khoj-ai/khoj into features/allow-multi-outputs-in-chat 2024-12-08 10:57:16 -08:00
sabaimran
19832a3ed0 Add note to uncomment line when using the prod image 2024-12-05 18:16:01 -08:00
sabaimran
110c64ba27 Update the desktop instructions 2024-12-05 17:39:16 -08:00
Debanjum
65c5b163c9 Add khoj_lantern svg to web public assets for use by new admin panel 2024-12-05 10:57:18 -08:00
Debanjum
354dc12b3b Style the Admin Panel with a modern theme and Khoj branding (#999) 
Overview
- The default django admin panel UI looks pretty dated and didn't
  have any Khoj specific branding
- Used the Unfold Django admin panel theme for a modern look
- Used the Khoj logo and name in Admin panel title, headings, favicons

Details:
All models shown on Admin panel need to inherit from unfold's
ModelAdmin to get styling applied. So

- Make all models on Admin panel inherit from unfold's ModelAdmin
- Subclassed UserAdmin to inherit from unfold's ModelAdmin
- Deregistered the unused Auth Group model from the Admin panel
  We can add it back when its actually used. Avoid confusion for now
- Explicitly register DjangoJobExecution on admin panel and again make
  it inherit from the unfold.admin.ModelAdmin
 2024-12-04 23:53:43 -08:00
Matias Forbord
9cc79c0fb7 Fix broken doc links to query filters from emacs docs page (#1000) 
* docs: repair query filters link

* change docs link to be relative
 2024-12-04 23:52:27 -08:00
sabaimran
8953ac03ec Rename additional context for llm response to program execution context 2024-12-04 18:43:41 -08:00
sabaimran
886fe4a0c9 Merge branch 'master' of github.com:khoj-ai/khoj into features/allow-multi-outputs-in-chat 2024-12-03 21:37:00 -08:00
sabaimran
df5e34615a Fix processing of images field when construct chat messages 2024-12-03 21:26:55 -08:00
sabaimran
3552032827 Rename additional context to additional_context_for_llm_response 2024-12-03 21:23:15 -08:00
sabaimran
d507894546 Simplify the desktop app 
- Make the desktop app mainly a file-syncing client for users who have lots of documents that they need to share with Khoj. This is because the web app provides a fairly robust chat client which can be used by anyone on their computer.
- The chat client in the desktop app had significantly drifted from our current brand / them, and didn't provide enough value add to update. Later, we will make it easier to install the existing web app as a desktop PWA.
 2024-12-02 15:54:05 -08:00
Debanjum
9f7cb335c5 Create Android app for Khoj (#991) 
Use bubblewrap to publish the Khoj Progressive Web App (PWA) as a Trusted Web Activity (TWA) to the Android Play Store
 2024-12-02 15:45:04 -08:00
Debanjum
ee28d7f125 Fix Android Studio build warnings by using newer gradle, mavenCentral 2024-12-02 12:48:33 -08:00
Debanjum
5d6fb07066 Fix app icons, orientation. Improve name, id, description in webmanifest 
- Use bubblewrap generated splash screen, notification icons from
  1200x1200 high res khoj icon in assets.khoj.dev.
- Discard bubblewrap generated launcher icons.
- Fix orientation to respect phone orientation settings. "any" was not.
 2024-12-02 12:48:25 -08:00
Debanjum
147c8e9115 Release v3 with high-res splash screen. More details in web, app manifest 
- Add 512, 192 Khoj maskable icons to web app manifest for android rendering
- Add id, categories etc suggested by pwabuilder
- Use higher quality icon images for splash screen than what
  bubblewrap creates by default
 2024-12-02 11:28:35 -08:00
Debanjum
d333e10e64 Encode request params as utf-8 to fix multibyte char error in khoj.el 
Encode api key in header, POST request body and GET query param for
search as utf-8 to avoid the multibyte char in request issue when
making API calls from khoj.el to khoj server.

Resolves #935
 2024-12-02 02:00:14 -08:00
Debanjum
db29894038 Do not wrap filepath in Path to fix indexing markdown files on Windows (#993) 
### Issue
- Path with / are converted to \\ on Windows using the `Path' operator.
- The `markdown_to_entries' module was trying to normalize file paths with`Path'  for some reason.
  This would store the file paths in DB Entry differently than the file to entries map if Khoj ran on Windows.
  That'd result in a KeyError when trying to look up the entry file path from `file_to_text_map' in the `text_to_entries:update_embeddings()' function.

### Fix
- Removing the unnecessary OS dependent Path normalization in `markdown_to_entries' should keep the file path storage consistent across `file_to_text_map' var, `FileObjectAdaptor', `Entry' DB tables on Windows for Markdown files as well.

This issue will affect users hosting Khoj server on Windows and attempting to index markdown files.

Resolves #984
 2024-12-02 01:02:58 -08:00
Debanjum
47c926b0ff Add more typing to org|md_to_entries. Remove redundant f-string wraps 
- Add type hints to improve maintainability of stabilzed indexing code
- It shouldn't be necessary to wrap string variables in an f-string

This change aims to improve code quality. It should not affect
functionality.
 2024-12-01 23:02:52 -08:00
Debanjum
dffdd81345 Do not wrap filepath in Path to fix indexing markdown files on Windows 
Issue
- Path with / are converted to \\ on Windows using the Path operator.
- The markdown to entries method for some reason was doing this.
  This would store the file paths in DB entry differently than the file
  to entries map. Resulting in a KeyError when trying to look up the
  entry file path from file_to_text_map in the
  text_to_entries:update_embeddings() function.

Fix
- Removing the unnecessary OS dependendent Path normalization in
  markdown_to_entries should keep the file path storage consistent
  across file_to_text_map var, FileObjectAdaptor, Entry DB tables on
  Windows for Markdown files as well

This issue would only affect users hosting Khoj server on Windows and
attempting to index markdown files.

Resolves #984
 2024-12-01 23:00:31 -08:00
sabaimran
c87fce5930 Add a migration to use the new image storage format for past conversations 
- Added it to the Django migrations so that it auto-triggers when someone updates their server and starts it up again for the first time. This will require that they update their clients as well in order to view/consume image content.
- Remove server-side references in the code that allow to parse the text-to-image intent as it will no longer be necessary, given the chat logs will be migrated
 2024-12-01 18:35:31 -08:00
Debanjum
9e0a2c7a98 Restrict generated chat title to 200 chars limit allowed for chat slug 2024-11-30 19:12:03 -08:00
Debanjum
8b8e2be82d Only create subscription object when it does not exist for user 
This avoid unnecessarily throwing an internal server error when the
user tries to sign-up using multiple mechanisms (e.g first by email, then
by google oauth)
 2024-11-30 19:08:34 -08:00
Debanjum
fc6be543bd Improve GPQA eval prompt to imrpove parsing answer from Khoj response 2024-11-30 17:21:09 -08:00
sabaimran
00f48dc1e8 If in the new images format, show the response text in obsidian instead of the inferred query 2024-11-30 14:39:51 -08:00
sabaimran
224abd14e0 Only add the image_url to the constructed chat message if it is a url 2024-11-30 14:39:27 -08:00
sabaimran
991577aa17 Allow a None turnId to accommodate historic chat messages 2024-11-30 14:39:08 -08:00
sabaimran
a539761c49 Fix processing of excalidrawdiagram in json response chunking 2024-11-30 12:35:13 -08:00
sabaimran
dc4a9ee3e1 Ensure that the generated assets are maintained in the chat window after streaming is completed. 2024-11-30 12:31:20 -08:00
sabaimran
e3aee50cf3 Fix parsing of generated_asset response 2024-11-29 18:41:53 -08:00
sabaimran
2b32f0e80d Remove commented out code blocks 2024-11-29 18:11:50 -08:00
sabaimran
df855adc98 Update response handling in Obsidian to work with new format 2024-11-29 18:10:47 -08:00
sabaimran
512cf535e0 Collapse train of thought when completed during live stream 2024-11-29 18:10:35 -08:00
sabaimran
a0b00ce4a1 Don't include null attributes when filling in stored conversation metadata 
- Prompt adjustments to indicate to LLM what context it has
 2024-11-29 18:10:14 -08:00
sabaimran
c5329d76ba Merge branch 'master' of github.com:khoj-ai/khoj into features/allow-multi-outputs-in-chat 2024-11-29 14:12:03 -08:00
sabaimran
46f647d91d Improve image rendering for khoj generated images. FIx typing of stored excalidraw image. 2024-11-29 14:11:48 -08:00
Debanjum
fdf69b7049 Publish second version with new upload key 2024-11-28 22:04:10 -08:00
Debanjum
faf15072b6 Create first version of Khoj Android app from PWA using Bubblewrap 2024-11-28 22:04:10 -08:00
sabaimran
4f6d1211ba Fix additional context type in anthropic chat 2024-11-28 20:16:36 -08:00
sabaimran
6f408948d3 Fix typing of generated_fiels parameters 2024-11-28 20:15:10 -08:00
sabaimran
439b18c21f Release Khoj version 1.30.10 2024-11-28 19:43:06 -08:00
sabaimran
2dfd163430 Add more explicity run strategies in the runner matrix 2024-11-28 19:31:34 -08:00
sabaimran
80cd902c86 Since linux/amd64 images aren't being created, try setting a custom description on the image 
Refer to this GH documentation on working with multi arch images in the container registry:
https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry#adding-a-description-to-multi-arch-images
 2024-11-28 19:14:06 -08:00
sabaimran
40d8a7a581 Release Khoj version 1.30.9 2024-11-28 18:45:50 -08:00
sabaimran
87aa653c7f Add additional steps in prod.Dockerfile to ensure dependencies are copied over 2024-11-28 18:37:08 -08:00
sabaimran
d91935c880 Initial commit of a functional but not yet elegant prototype for this concept 2024-11-28 17:28:23 -08:00
Debanjum
a552543f4f Use json5 to parse llm generated questions to query docs and web 
json5 is more forgiving, handles double quotes, newlines in raw json
string
 2024-11-28 14:35:34 -08:00
Debanjum
0a69af4f61 Update to latest ToDesktop runtime 2024-11-28 13:56:14 -08:00
Debanjum
1d0fe141dc Release Khoj version 1.30.8 2024-11-28 13:37:30 -08:00
Debanjum
29e801c381 Add MATH500 dataset to eval 
Evaluate simpler MATH500 responses with gemini 1.5 flash

This improves both the speed and cost of running this eval
 2024-11-28 12:48:25 -08:00
Debanjum
22aef9bf53 Add GPQA (diamond) dataset to eval 2024-11-28 12:48:25 -08:00
Debanjum
f1190ccf32 Improve parsing complex json strings returned by LLM (#989) 
- Improve escaping to load complex json objects
- Fallback to a more forgiving [json5](https://json5.org/) loader if `json.loads` cannot parse complex json str

This should reduce failures to pick research tool and run code by agent
 2024-11-28 11:01:39 -08:00
Debanjum
8c120a5139 Fallback to json5 loader if json.loads cannot parse complex json str 
JSON5 spec is more flexible, try to load using a fast json5 parser if
the stricter json.loads from the standard library can't load the
raw complex json string into a python dictionary/list
 2024-11-26 21:17:00 -08:00
Debanjum
70b7e7c73a Improve load of complex json objects. Use it to pick tool, run code 
Gemini doesn't work well when trying to output json objects. Using it
to output raw json strings with complex, multi-line structures
requires more intense clean-up of raw json string for parsing
 2024-11-26 17:37:57 -08:00
Debanjum
8cb0db0051 Fix llama-cpp-python install by pytest github workflow 
- Use pre-built wheels for torch and llama-cpp-python
- Install and link musl as it's used by llama-cpp-python pre-built
  wheel instead of glibc
- Join Install git and Install Dependencies steps in pytest workflow
  To remove unnecessary steps
 2024-11-26 02:04:36 -08:00
Debanjum
29315f44e7 Add assetlinks.json to link android app to app.khoj.dev domain 
Add sha cert of android upload, signing keys to open debug, prod apps
as TWA in fullscreen on android phones
 2024-11-26 01:57:54 -08:00
Debanjum
a97a45bf20 Align agent personality with recently updated khoj personality 
See update to Khoj personality in commit
6eb59464da
 2024-11-26 00:06:16 -08:00
Debanjum
e088fcbc7b Build for arm64 on arm64 runner. Parallelize arm64, x64 docker builds 
- Building arm64 image on an ubuntu arm64 runner reduces `yarn build'
  step time by 75% from 12mins to 3mins.
  - This is because no QEMU emulation for arm64 on x86 is required now
- Parallelizing x64 and arm64 platform builds halves build time on top
  - Revert to use standard ubuntu-latest runner as large x64 runner
    doesn't give much more speed improvements

This results an effective additional 50%-66% reduction in build time
on top of #987.

So a full dockerize workflow run now takes *10 mins* vs previous 35+mins.
This is a total of *72% improvement* in max dockerize run time.

Get additional speed improvements when docker layer cache hit.
 2024-11-24 23:18:55 -08:00
Debanjum
5723a3778e Speed up Docker image builds using multi-stage parallel pipelines (#987) 
## Objective
Improve build speed and size of khoj docker images

## Changes
### Improve docker image build speeds
  - Decouple web app and server build steps
  - Build the web app and server in parallel
  - Cache docker layers for reuse across dockerize github workflow runs
    - Split Docker build layers for improved cacheability (e.g separate `yarn install` and `yarn build` steps)
### Reduce size of khoj docker images 
  - Use an up-to-date `.dockerignore` to exclude unnecessary directories
  - Do not installing cuda python packages for cpu builds
### Improve web app builds
  - Use consistent mechanism to get fonts for web app
  - Make tailwind extensions production instead of dev dependencies
  - Make next.js create production builds for the web app (via `NODE_ENV=production` env var)
 2024-11-24 21:49:46 -08:00
Debanjum
4a5646c8da Cache docker layers, nextjs builds in dockerize github workflow 2024-11-24 21:06:22 -08:00
Debanjum
6a39651ad3 Standardize loading fonts locally across pages on web app 2024-11-24 20:41:15 -08:00
sabaimran
9368699b2c Migrate the pre-commit config 2024-11-24 14:54:26 -08:00
sabaimran
6eb59464da Add additional reinforcement to coax gemini into giving a minimum helpful response 2024-11-24 14:53:53 -08:00
sabaimran
15f062b34a Remove print statement for agent style map 2024-11-24 14:53:53 -08:00
sabaimran
d7e68a2d1b Wait for iplcodata to load before first message 
- Fix the console khoj ai ascii art
- Remove some not so good suggested prompt
 2024-11-24 14:53:53 -08:00
Debanjum
f51e0f7859 Make Next.js create production builds of web app for Docker images 2024-11-24 13:59:40 -08:00
Debanjum
710e00ad9e Make tailwind extensions prod, instead of dev, deps of web app 2024-11-24 13:59:40 -08:00
Debanjum
4b486ea5f6 Exclude unnecessary directories from final docker builds 2024-11-24 13:59:40 -08:00
Debanjum
78d8ca49ec Skip Nvidia GPU python packages during Server install in Dockerfiles 2024-11-24 13:59:39 -08:00
Debanjum
37887a175a Speed up Docker image builds using multi-stage parallel pipelines 
Decouple web app, server builds in parallel to speed up Docker
builds
 2024-11-24 12:48:30 -08:00
Debanjum
7c77d65d35 Improve logic to disable telemetry via KHOJ_TELEMETRY_DISABLE env var 
The newly added KHOJ_TELEMETRY_DISABLE env var knob to disable
telemetry should override old config mechanism when set
 2024-11-24 00:54:16 -08:00
sabaimran
2d683898c2 Release Khoj version 1.30.7 2024-11-23 22:51:10 -08:00
sabaimran
914ff994f7 Fix cost addition to chat_metadata 2024-11-23 22:50:45 -08:00
Debanjum
caaa127dcf Release Khoj version 1.30.6 2024-11-23 21:07:00 -08:00
Debanjum
57b8273002 Fix apt install for musl-dev in prod.Dockerfile 2024-11-23 21:06:09 -08:00
Debanjum
8f966b11ec Release Khoj version 1.30.5 2024-11-23 20:49:05 -08:00
Debanjum
498895a47d Fix libmusl error using pre-built llama-cpp-python wheel in prod Docker 2024-11-23 20:47:41 -08:00
Debanjum
e5b211a743 Release Khoj version 1.30.4 2024-11-23 19:48:21 -08:00
Debanjum
9848d89d03 Try build docker images with github high cpu, ram runner 2024-11-23 19:09:36 -08:00
Debanjum
04bb3d6f15 Fix libmusl error using pre-built llama-cpp-python wheel via Docker 
Seems like llama-cpp-python pre-built wheels need libmusl. Otherwise
you run into runtime errors on Khoj startup via Docker.
 2024-11-23 18:46:44 -08:00
Debanjum
8dd2122817 Set sample size to 200 for automated eval runs as well 2024-11-23 14:48:38 -08:00
Debanjum
c4ef31d86f Release Khoj version 1.30.3 2024-11-23 14:40:06 -08:00
Debanjum
15ae22bdcf Use pre-built llama-cpp-python wheel in Khoj docker images 
Reduces build time and resolves FileNotFoundError 'ninja' during
llama-cpp-python local build.
 2024-11-23 14:38:07 -08:00
sabaimran
4ac49ca90f Release Khoj version 1.30.2 2024-11-23 12:00:28 -08:00
sabaimran
eb1b21baaa Add a new sign in modal that is triggered from the login prompt screen, rather than redirecting user to another screen to sign in 2024-11-23 11:55:34 -08:00
Debanjum
5aa5cb1941 Add "New" section with latest updates to Readme 2024-11-23 01:36:50 -08:00
sabaimran
7f5bf35806 Disambiguate renewal_date type. Previously, being used as None, False, and Datetime in different places. 2024-11-22 12:06:20 -08:00
sabaimran
5e8c824ecc Improve the experience for finding past conversation 
- add a conversation title search filter, and an agents filter, for finding conversations
- in the chat session api, return relevant agent style data
 2024-11-22 12:03:01 -08:00
sabaimran
a761865724 Fix handling of customer.subscription.updated event to process new renewal end date 2024-11-22 12:03:01 -08:00
sabaimran
6a054d884b Add quicker/easier filtering on auth 2024-11-22 12:03:01 -08:00
Debanjum
b9a889ab69 Fix Khoj responses when code generated charts in response context 
The current fix should improve Khoj responses when charts in response
context. It truncates code context before sharing with response chat actors.

Previously Khoj would respond with it not being able to create chart
but than have a generated chart in it's response in default mode.

The truncate code context was added to research chat actor for
decision making but it wasn't added to conversation response
generation chat actors.

When khoj generated charts with code for its response, the images in
the context would exceed context window limits.

So the truncation logic to drop all past context, including chat
history, context gathered for current response.

This would result in chat response generator 'forgetting' all for the
current response when code generated images, charts in response context.
 2024-11-21 14:43:52 -08:00
Debanjum
5475a262d4 Move truncate code context func for reusability across modules 
It needs to be used across routers and processors. It being in
run_code tool makes it hard to be used in other chat provider contexts
due to circular dependency issues created by
send_message_to_model_wrapper func
 2024-11-21 14:27:39 -08:00
Debanjum
f434c3fab2 Fix toggling prompt tracer on/off in Khoj via PROMPTRACE_DIR env var 
Previous changes to depend on just the PROMPTRACE_DIR env var instead
of KHOJ_DEBUG or verbosity flag was partial/incomplete.

This fix adds all the changes required to only depend on the
PROMPTRACE_DIR env var to enable/disable prompt tracing in Khoj.
 2024-11-21 14:06:00 -08:00
Debanjum
4a40cf79c3 Add docs on how to cross-device access self-hosted khoj using tailscale 2024-11-21 11:07:18 -08:00
Debanjum
1f96c13f72 Enable starting khoj uvicorn server with ssl cert file, key for https 
Pass your domain cert files via the --sslcert, --sslkey cli args.
For example, to start khoj at https://example.com, you'd run command:

KHOJ_DOMAIN=example.com khoj --sslcert example.com.crt --sslkey
example.com.key --host example.com

This sets up ssl certs directly with khoj without requiring a
reverse proxy like nginx to serve khoj behind https endpoint for
simple setups. More complex setups should, of course, still use a
reverse proxy for efficient request processing
 2024-11-21 11:07:18 -08:00
sabaimran
9fea02f20f In telemetry, differentiate create_user google and email 2024-11-21 11:01:37 -08:00
sabaimran
9db885b5f7 Limit access to chat models to futurist users 2024-11-21 07:53:24 -08:00
sabaimran
7a00a07398 Add trailing slash to Ollama url in docs 2024-11-21 07:48:18 -08:00
sabaimran
3519dd76f0 Fix type of excalidraw image response 2024-11-20 19:01:13 -08:00
sabaimran
467de76fc1 Improve the image diagramming prompts and response parsing 2024-11-20 18:59:40 -08:00
Debanjum
50d8405981 Enable khoj to use terrarium code sandbox as tool in eval workflow 2024-11-20 14:19:27 -08:00
Debanjum
2203236e4c Update desktop app dependencies 2024-11-20 13:05:55 -08:00
Debanjum
409204917e Update documentation website dependencies 2024-11-20 13:05:32 -08:00
Debanjum
6f1adcfe67 Track Usage Metrics in Chat API. Track Running Cost, Accuracy in Evals (#985) 
- Track, return cost and usage metrics in chat api response
  Track input, output token usage and cost of interactions with 
  openai, anthropic and google chat models for each call to the khoj chat api
- Collect, display and store costs & accuracy of eval run currently in progress
  This provides more insight into eval runs during execution 
  instead of having to wait until the eval run completes.
 2024-11-20 12:59:44 -08:00
Debanjum
ffbd0ae3a5 Fix eval github workflow to run on releases, i.e on tags push 2024-11-20 12:57:42 -08:00
Debanjum
ed364fa90e Track running costs & accuracy of eval runs in progress 
Collect, display and store running costs & accuracy of eval run.

This provides more insight into eval runs during execution instead of
having to wait until the eval run completes.
 2024-11-20 12:40:51 -08:00
Debanjum
bbd24f1e98 Improve dropdown menus on web app setting page with scroll & min-width 
- Previously when settings list became long the dropdown height would
  overflow screen height. Now it's max height is clamped and  y-scroll
- Previously the dropdown content would take width of content. This
  would mean the menu could sometimes be less wide than the button. It
  felt strange. Now dropdown content is at least width of parent button
 2024-11-20 12:27:13 -08:00
Debanjum
c53c3db96b Track, return cost and usage metrics in chat api response 
- Track input, output token usage and cost for interactions
  via chat api with openai, anthropic and google chat models

- Get usage metadata from OpenAI using stream_options
- Handle openai proxies that do not support passing usage in response

- Add new usage, end response  events returned by chat api.
  - This can be optionally consumed by clients at a later point
  - Update streaming clients to mark message as completed after new
    end response event, not after end llm response event
- Ensure usage data from final response generation step is included
  - Pass usage data after llm response complete. This allows gathering
    token usage and cost for the final response generation step across
    streaming and non-streaming modes
 2024-11-20 12:17:58 -08:00
Debanjum
80df3bb8c4 Enable prompt tracing only when PROMPTRACE_DIR env var set 
Better to decouple prompt tracing from debug mode or verbosity level
and require explicit, independent config to enable prompt tracing
 2024-11-20 11:54:02 -08:00
Debanjum
9ab76ccaf1 Skip adding agent to chat metadata when chat unset to avoids null ref 2024-11-19 21:10:23 -08:00
Debanjum
4da0499cd7 Stream responses by openai's o1 model series, as api now supports it 
Previously o1 models did not support streaming responses via API. Now
they seem to do
 2024-11-19 21:10:23 -08:00
sabaimran
e5347dac8c Fix base image used for prod in docs 2024-11-19 15:51:27 -08:00
sabaimran
b943069577 Fix button text, and login url in self-hosted auth docs 2024-11-19 15:50:13 -08:00
sabaimran
3b5e6a9f4d Update authentication documentation 2024-11-19 15:45:47 -08:00
Debanjum
7bdc9590dd Fix handling sources, output in chat actor when is automated task 
Remove unnecessary ```python prefix removal. It isn't being triggered
in json deserialize path.
 2024-11-19 13:49:27 -08:00
Debanjum
0e7d611a80 Remove ```python codeblock prefix from raw json before deserialize 2024-11-19 12:53:52 -08:00
Debanjum
001c13ef43 Upgrade web app package dependencies 2024-11-19 12:53:52 -08:00
sabaimran
4f5c1eeded Update some of the open graph data for the documentation website 2024-11-19 11:14:46 -08:00
sabaimran
5134d49d71 Release Khoj version 1.30.1 2024-11-18 17:30:33 -08:00
sabaimran
8bdd0b26d3 And a connections clean up decorator to all scheduled tasks 2024-11-18 17:19:36 -08:00
Debanjum
817601872f Update default offline models enabled 2024-11-18 16:38:17 -08:00
Debanjum
45c623f95c Dedupe, organize chat actor, director tests 
- Move Chat actor tests that were previously in chat director tests file
- Dedupe online, offline io selector chat actor tests
 2024-11-18 16:10:50 -08:00
Debanjum
2a76c69d0d Run online, offine chat actor, director tests for any supported provider 
- Previously online chat actors, director tests only worked with openai.
  This change allows running them for any supported onlnie provider
  including Google, Anthropic and Openai.

- Enable online/offline chat actor, director in two ways:
  1. Explicitly setting KHOJ_TEST_CHAT_PROVIDER environment variable to
     google, anthropic, openai, offline
  2. Implicitly by the first API key found from openai, google or anthropic.

- Default offline chat provider to use Llama 3.1 3B for faster, lower
  compute test runs
 2024-11-18 15:11:37 -08:00
Debanjum
653127bf1d Improve data source, output mode selection 
- Set output mode to single string. Specify output schema in prompt
  - Both thesee should encourage model to select only 1 output mode
    instead of encouraging it in prompt too many times
  - Output schema should also improve schema following in general
- Standardize variable, func name of io selector for readability
- Fix chat actors to test the io selector chat actor
- Make chat actor return sources, output separately for better
  disambiguation, at least during tests, for now
 2024-11-18 15:11:37 -08:00
Debanjum
e3fd51d14b Pass user arg to create title from query in new automation flow 2024-11-18 12:58:10 -08:00
Debanjum
9e74de9b4f Improve serializing conversation JSON to print messages on console 
- Handle chatml message.content with non-json serializable data like
  WebP image binary data used by Gemini models
 2024-11-18 12:57:05 -08:00
sabaimran
3f70d2f685 Add more graceful exception handling when tool selection doesn't work 2024-11-18 09:34:49 -08:00
Debanjum
a2ccf6f59f Fix github workflow to start Khoj, connect to PG and upload results 
- Do not trigger tests to run in ci on update to evals
 2024-11-18 04:25:15 -08:00
Debanjum
7c0fd71bfd Add GitHub workflow to quiz Khoj across modes and specified evals (#982) 
- Evaluate khoj on random 200 questions from each of google frames and openai simpleqa benchmarks across *general*, *default* and *research* modes
- Run eval with Gemini 1.5 Flash as test giver and Gemini 1.5 Pro as test evaluator models
- Trigger eval workflow on release or manually
- Make dataset, khoj mode and sample size configurable when triggered via manual workflow
- Enable Web search, webpage read tools during evaluation
 2024-11-18 02:19:30 -08:00
sabaimran
f75085dc7a Release Khoj version 1.30.0 2024-11-17 21:36:22 -08:00
sabaimran
c72813ba67 Merge pull request #981 from rznzippy/bugfix/980/database-connections-leakage 
Fix database connections leakage (#980)
 2024-11-17 21:01:06 -08:00
sabaimran
7d50c6590d Merge pull request #977 from khoj-ai/features/improve-tool-selection 
- JSON extract from LLMs is pretty decent now, so get the input tools and output modes all in one go. It'll help the model think through the full cycle of what it wants to do to handle the request holistically.
- Make slight improvements to tool selection indicators
 2024-11-17 20:08:19 -08:00
sabaimran
282f47e0d6 Add Jina documentation to readme for self-hosting 2024-11-17 17:20:28 -08:00
Debanjum
48567fd468 Do not erase partial message when generation stopped via button on web app 
Previously, we'd replace the generated message with an error message
when message generation stopped via stop button on chat page of web app.
So the partially generated message (which could be useful) gets lost.

This change just stops generation, while keeping the generated
response so any useful information from the partially generated
message can be retrieved.
 2024-11-17 16:29:18 -08:00
Debanjum
285006d6c9 Sync chat models in Khoj with OpenAI proxies (e.g Ollama) on startup 
- Allows managing chat models in the OpenAI proxy service like Ollama.
- Removes need to manually add, remove chat models from Khoj Admin Panel
  for these OpenAI compatible API services when enabled.
- Khoj still mantains the chat models configs within Khoj, so they can
  be configured via the Khoj admin panel as usual.
 2024-11-17 15:34:36 -08:00
Debanjum
4a7f5d1abe Set API keys in docker-compose.yml to enable web search, scrape tools 2024-11-17 15:34:36 -08:00
Debanjum
d6eece63f4 Use Jina API Key of Jina web scraper if configured in DB 
Previously Jina search didn't API key. Now that it does need API key,
we should re-use the API key set in the Jina web scraper config,
otherwise fallback to using JINA_API_KEY from environment variable, if
either is present.

Resolves #978
 2024-11-17 15:34:14 -08:00
sabaimran
6531f24ca0 Further improvements for descriptions to LLM on modes, code, diagram, image. 2024-11-17 13:23:57 -08:00
sabaimran
0eba6ce315 When diagram generation fails, save to conversation log 
- Update tool name when choosing tools to execute
 2024-11-17 13:23:12 -08:00
sabaimran
7e662a05f8 Merge branch 'master' of github.com:khoj-ai/khoj into features/improve-tool-selection 2024-11-17 12:26:55 -08:00
Ilya Khrustalev
00b1af8f99 Fix database connections leakage (#980) 2024-11-17 19:15:05 +01:00
Debanjum
69ef6829c1 Simplify integrating Ollama, OpenAI proxies with Khoj on first run 
- Integrate with Ollama or other openai compatible APIs by simply
  setting `OPENAI_API_BASE' environment variable in docker-compose etc.
- Update docs on integrating with Ollama, openai proxies on first run
- Auto populate all chat models supported by openai compatible APIs
- Auto set vision enabled for all commercial models

- Minor
  - Add huggingface cache to khoj_models volume. This is where chat
  models and (now) sentence transformer models are stored by default
  - Reduce verbosity of yarn install of web app. Otherwise hit docker
  log size limit & stops showing remaining logs after web app install
  - Suggest `ollama pull <model_name>` to start it in background
 2024-11-17 02:08:20 -08:00
Debanjum
2366fa08b9 Update default vision supported & anthropic chat models on first run 
- Update to latest initialize with new claude 3.5 sonnet and haiku models
- Update to set vision enabled for google and anthropic models by
  default. Previously we didn't support but we've supported this for a
  month or two now
 2024-11-17 02:08:20 -08:00
Debanjum
23ab258d78 Improve user conversation config details on Admin panel 
Show user email and chat model that is associated with the user
conversation config
 2024-11-17 02:08:20 -08:00
Debanjum
41d9011a26 Move evaluation script into tests/evals directory 
This should give more space for eval scripts, results and readme
 2024-11-17 02:08:20 -08:00
Debanjum
d9d5884958 Enable evaluating Khoj on the OpenAI SimpleQA bench using eval script 
- Just load the raw csv from OpenAI bucket. Normalize it into FRAMES format
- Improve docstring for frames datasets as well
- Log the load dataset perf timer at info level
 2024-11-17 02:08:20 -08:00
Debanjum
eb5bc6d9eb Remove Talc search bench from Khoj eval script 2024-11-17 02:08:20 -08:00
Debanjum
fc45aceecf Delete unused favicon ico in old web app directory 2024-11-17 02:08:20 -08:00
Debanjum
a16fc3ade8 Only add /research prefix when no slash command in message on web app 
- Explictly adding a slash command is a higher priority intent than
  research mode being enabled in the background. Respect that for a
  more intuitive UX flow.

- Explicit slash commands do not currently work in research mode.
  You've to turn research mode off to use other slash commands. This
  is strange, unnecessary given intent priority is clear.
 2024-11-17 02:08:20 -08:00
sabaimran
a1b4587b34 Remove extract_images flag from PDF loader 2024-11-15 21:46:35 -08:00
sabaimran
15b4cec1e8 Add documentation for how to use the text to image model configs, reduce to Replicate 2024-11-15 15:26:14 -08:00
sabaimran
759873ec44 Add documentation for how to use the text to image model configs 2024-11-15 15:22:06 -08:00
sabaimran
c77dc84a68 Remove output_modes function reference in chat tests 2024-11-15 14:03:07 -08:00
sabaimran
e3f1ea9dee Improve tool, output mode selection process 
- JSON extract from LLMs is pretty decent now, so get the input tools and output modes all in one go. It'll help the model think through the full cycle of what it wants to do to handle the request holistically.
- Make slight improvements to tool selection indicators
 2024-11-15 13:53:53 -08:00
sabaimran
c1a5b32ebf Do not start server when importing the main.py file, unless gunicorn 
- Add more graceful shutdown when closing bg scheduler thread
 2024-11-14 17:36:51 -08:00
sabaimran
be3ee5ec9f Add cool new suggestion cards for math, diagramming 2024-11-14 17:36:51 -08:00
Debanjum
9fc44f1a7f Enable evaluation Khoj on the Talc Search Bench using Eval script 
- Just load the raw jsonl from Github and normalize it into FRAMES format
- Color printed accuracy in eval script to blue for readability
 2024-11-13 22:50:14 -08:00
Debanjum
8e009f48ce Show tool call error in next iteration. Allow rerun if model requests. 
Previously errors would get eaten up but the model wouldn't see
anything. And the model wouldn't be allowed re-run the same query-tool
combination in the next iteration.

This update should give it insight into why it didn't get a result. So
it can make an informed (hopefully better) decision on what to do next.
And re-run the previous query if appropriate.
 2024-11-13 22:50:14 -08:00
Debanjum
604da90fa8 Wrap try/catch around online search in research mode like other tools 
Previously when call to online search API etc. failed, it'd error out
of response to query in research mode. Khoj should skip tool use that
iteration but continue to try respond.
 2024-11-13 16:46:09 -08:00
Debanjum
8851b5f78a Standardize chat message truncation and serialization before print 
Previously chatml messages were just strings, now they can be list of
strings or list of dicts as well.

- Use json seriallization to manage their variations and truncate them
  before printing for context.
- Put logic in single function for use across chat models
 2024-11-13 16:30:17 -08:00
Debanjum
f4e37209a2 Improve error handling, display and configurability of eval script 
- Default to evaluation decision of None when either agent or
  evaluator llm fails. This fixes accuracy calculations on errors
- Fix showing color for decision True
- Enable arg flags to specify output results file paths
 2024-11-13 14:32:22 -08:00
Debanjum
15b0cfa3dd Improve structured message truncation in logger 
Previously chatml messages were just strings.
Since gemini, anthropic models always have messages as list of
strings, truncate those strings instead of the list of message content
 2024-11-13 14:32:22 -08:00
Debanjum
153ae8bea9 Cut binary, long output files from code result for context efficiency 
Removing binary data and truncating large data in output files
generated by code runs should improve speed and cost of research mode
runs with large or binary output files.

Previously binary data in code results was passed around in iteration
context during research mode. This made the context inefficient
because models have limited efficiency and reasoning capabilities over
b64 encoded image (and other binary) data and would hit context limits
leading to unnecessary truncation of other useful context

Also remove image data when logging output of code execution
 2024-11-13 14:32:22 -08:00
sabaimran
de34cc3987 Remove og image url with khoj documentation 2024-11-13 10:23:02 -08:00
sabaimran
4a1b1e8b9a Add support for interrupting messages after they've been sent. 2024-11-12 22:22:45 -08:00
sabaimran
d607ad7a27 Release Khoj version 1.29.1 2024-11-12 10:32:56 -08:00
sabaimran
8ec1764e42 Handle size calculation more gracefully for converted documents, depending on type 2024-11-12 02:00:29 -08:00
sabaimran
b6714c202f Increase the title character limit to 500 for conversations 2024-11-12 01:51:19 -08:00
sabaimran
f05e64cf8c Release Khoj version 1.29.0 2024-11-11 21:46:25 -08:00
sabaimran
47d3c8c235 Remove email query parameter from subscription patch api 2024-11-11 21:39:49 -08:00
sabaimran
d7027109a5 And null handling for response output_files in code output 2024-11-11 21:14:56 -08:00
sabaimran
d68243a3fb Revert clean_json logic temporarily. Eventually, we should do better validation here to extract markdown-formatted json. 2024-11-11 21:05:17 -08:00
sabaimran
1cab6c081f Add better error handling for diagram output, and fix chat history construct 
- Make the `clean_json` method more robust as well
 2024-11-11 20:44:19 -08:00
sabaimran
7bd2f83f97 Wrap test in suggestionCard 2024-11-11 20:12:46 -08:00
Debanjum
48862a8400 Enable Passing External Documents for Analysis in Code Sandbox (#960) 
- Allow passing user files as input into code sandbox for analysis
- Update prompt to give more example of complex, multi-line code
- Simplify logic for model. Run one program at a time, 
  instead of allowing model to run multiple programs in parallel
- Show Code generated charts and docs in Reference pane of web app and make them downloaded
 2024-11-11 19:37:17 -08:00
Debanjum
5078ac0ce2 Await on conversation save when generate conversation title via API 2024-11-11 19:17:39 -08:00
Debanjum
e1d0015248 Allow disabling Khoj telemetry via KHOJ_TELEMETRY_DISABLE env var 2024-11-11 19:17:39 -08:00
Debanjum
a52500d289 Show generated code artifacts before notes and online references 2024-11-11 18:00:22 -08:00
Debanjum
218eed83cd Show output file not code on hover. Remove reference card title border 2024-11-11 18:00:22 -08:00
Debanjum
b970cfd4b3 Align styling of reference panel card across code, docs, web results 
- Add a border below heading
- Show code snippet in pre block
- Overflow-x when reference side panel open to allow seeing whole text
  via x-scroll
- Align header, body position of reference cards with each other
- Only show filename in doc reference cards at message bottom.
  Show full file path in hover and reference side panel
 2024-11-11 18:00:22 -08:00
Debanjum
8e9f4262a9 Render code output files with code references in reference section 
- Improve rendering code reference with better icons, smaller text and
  different line clamps for better visibility
- Show code output files as sub card of code card in reference section
- Allow downloading files generated by code instead of rendering it in
  chat message directly
- Show executed code before online references in reference panel
 2024-11-11 18:00:22 -08:00
Debanjum
92c1efe6ee Fixes to render & save code context with non text based output modes 
- Fix to render code generated chart with images, excalidraw diagrams
- Fix to save code context to chat history in image, diagram output modes
- Fix bug in image markdown being wrapped twice in markdown syntax
- Render newline in code references shown on chat page of web app
  Previously newlines weren't getting rendered. This made the code
  executed by Khoj hard to read in references. This changes fixes that.

  `dangerouslySetInnerHTML' usage is justified as rendered code
  snippet is being sanitized by DOMPurify before rendering.
 2024-11-11 18:00:22 -08:00
Debanjum
af0215765c Decode code text output files from b64 to str to ease client processing 2024-11-11 18:00:22 -08:00
Debanjum
7b39f2014a Enable analysing user documents in code sandbox and other improvements 
- Run one program at a time, instead of allowing model to pass
  multiple programs to run in parallel to simplify logic for model
- Update prompt to give more example of complex, multi-line code
- Allow passing user files as input into code sandbox for analysis

- Log code execution timer at info level to evaluate execution latencies
  in production

- Type the generated code for easier processing by caller functions
 2024-11-11 17:59:37 -08:00
sabaimran
dc109559d4 Research mode gray when off, colored when on 2024-11-11 16:35:07 -08:00
sabaimran
cdda9c2e73 Improve text wrapping for attached files and preview context 
For the research mode toggle, make it not fill when it's off
 2024-11-11 13:32:10 -08:00
sabaimran
dd36303bb7 Fix sending file attachments in save_to_conversation method 
- When files attached but upload fails, don't update the state variables
- Make removing null characters in pdf extraction more space efficient
 2024-11-11 12:53:06 -08:00
Debanjum
ba2471dc02 Do not CRUD on entries, files & conversations in DB for null user (#958) 
Increase defense-in-depth by reducing paths to create, read, update or delete entries, files and conversations in DB when user is unset.
 2024-11-11 12:47:22 -08:00
Debanjum
536fe994be Remove unused db adapter methods, like for fact checker data store 2024-11-11 12:22:34 -08:00
Debanjum
10bca6fa8f Convert required user param check into decorator. Use with more adapters 2024-11-11 12:22:32 -08:00
Debanjum
ff5c10c221 Do not CRUD on entries, files & conversations in DB for null user 
Increase defense-in-depth by reducing paths to create, read, update or
delete entries, files and conversations in DB when user is unset.
 2024-11-11 12:20:07 -08:00
sabaimran
27fa39353e Make custom agent creation flow available to everyone 
- For private agents, add guardrails to prevent against any misuse or violation of terms of service.
 2024-11-11 11:54:59 -08:00
sabaimran
b563f46a2e Merge pull request #957 from khoj-ai/features/include-full-file-in-convo-with-filter 
Support including file attachments in the chat message

Now that models have much larger context windows, we can reasonably include full texts of certain files in the messages. Do this when an explicit file filter is set in a conversation. Do so in a separate user message in order to mitigate any confusion in the operation.

Pipe the relevant attached_files context through all methods calling into models.

This breaks certain prior behaviors. We will no longer automatically be processing/generating embeddings on the backend and adding documents to the "brain". You'll have to go to settings and go through the upload documents flow there in order to add docs to the brain (i.e., have search include them during question / response).
 2024-11-11 11:34:42 -08:00
sabaimran
2bb2ff27a4 Rename attached_files to query_files. Update relevant backend and client-side code. 2024-11-11 11:21:26 -08:00
sabaimran
47937d5148 Merge branch 'features/include-full-file-in-convo-with-filter' of github.com:khoj-ai/khoj into features/include-full-file-in-convo-with-filter 2024-11-11 09:34:08 -08:00
sabaimran
ae4eb96d48 Consolidate file name to icon mapping 2024-11-11 09:34:04 -08:00
Debanjum
7954f39633 Use accept param to file input to indicate supported file types in web app 
Remove unused total size calculations in chat input
 2024-11-11 04:06:17 -08:00
Debanjum
4223b355dc Use python stdlib methods to write pdf, docx to temp files for loaders 
Use python standard method tempfile.NamedTemporaryFile to write,
delete temporary files safely.
 2024-11-11 03:24:50 -08:00
Debanjum
fd15fc1e59 Move construct chat history back to it's original position in file 
Keep function where it original was allows tracking diffs and change
history more easily
 2024-11-11 03:24:50 -08:00
Debanjum
35d6c792e4 Show snippet of truncated messages in debug logs to avoid log flooding 2024-11-11 02:30:38 -08:00
sabaimran
8805e731fd Merge branch 'master' of github.com:khoj-ai/khoj into features/include-full-file-in-convo-with-filter 2024-11-10 19:24:11 -08:00
sabaimran
a5e2b9e745 Exit early when running an automation if the conversation for the automation does not exist. 2024-11-10 19:22:21 -08:00
sabaimran
55200be4fa Apply agent color fill to the toggle both in off and on states 2024-11-10 19:16:43 -08:00
Debanjum
7468f6a6ed Deduplicate online references returned by chat API to clients 
This will ensure only unique online references are shown in all
clients.

The duplication issue was exacerbated in research mode as even with
different online search queries, you can get previously seen results.

This change does a global deduplication across all online results seen
across research iterations before returning them in client reponse.
 2024-11-10 16:10:32 -08:00
Debanjum
137687ee49 Deduplicate searches in normal mode & across research iterations 
- Deduplicate online, doc search queries across research iterations.
  This avoids running previously run online, doc searches again and
  dedupes online, doc context seen by model to generate response.
- Deduplicate online search queries generated by chat model for each
  user query.
- Do not pass online, docs, code context separately when generate
  response in research mode. These are already collected in the meta
  research passed with the user query
- Improve formatting of context passed to generate research response
  - Use xml tags to delimit context. Pass per iteration queries in each
    iteration result
  - Put user query before meta research results in user message passed
    for generating response

This deduplications will improve speed, cost & quality of research mode
 2024-11-10 16:10:32 -08:00
Debanjum
306f7a2132 Show error in picking next tool to researcher llm in next iteration 
Previously the whole research mode response would fail if the pick
next tool call to chat model failed. Now instead of it completely
failing, the researcher actor is told to try again in next iteration.

This allows for a more graceful degradation in answering a research
question even if a (few?) calls to the chat model fail.
 2024-11-10 14:52:02 -08:00
Debanjum
eb492f3025 Only keep webpage content requested, even if Jina API gets more data 
Jina search API returns content of all webpages in search results.
Previously code wouldn't remove content beyond max_webpages_to_read
limit set. Now, webpage content in organic results aree explicitly
removed beyond the requested max_webpage_to_read limit.

This should align behavior of online results from Jina with other
online search providers. And restrict llm context to a reasonable size
when using Jina for online search.
 2024-11-10 14:51:16 -08:00
Debanjum
8ef7892c5e Exclude non-dictionary doc context from chat history sent to chat models 
This fixes chat with old chat sessions. Fixes issue with old Whatsapp
users can't chat with Khoj because chat history doc context was
stored as a list earlier
 2024-11-10 14:51:16 -08:00
Debanjum
d892ab3174 Fix handling of command rate limit and improve rate limit messages 
Command rate limit wouldn't be shown to user as server wouldn't be
able to handle HTTP exception in the middle of streaming.

Catch exception and render it as LLM response message instead for
visibility into command rate limiting to user on client

Log rate limmit messages for all rate limit events on server as info
messages

Convert exception messages into first person responses by Khoj to
prevent breaking the third wall and provide more details on wht
happened and possible ways to resolve them.
 2024-11-10 14:51:16 -08:00
Debanjum
80ee35b9b1 Wrap messages in web, obsidian UI to stay within screen when long links 
Wrap long links etc. in chat messages and train of thought lists on
web app app and obsidian plugin by breaking them into newlines by word
 2024-11-10 14:49:51 -08:00
Debanjum
f967bdf702 Show correct example index being currently processed in frames eval 
Previously the batch start index wasn't being passed so all batches
started in parallel were showing the same processing example index

This change doesn't impact the evaluation itself, just the index shown
of the example currently being evaluated
 2024-11-10 14:49:51 -08:00
Debanjum
84a8088c2b Only evaluate non-empty responses to reduce eval script latency, cost 
Empty responses by Khoj will always be an incorrect response, so no
need to make call to an evaluator agent to check that
 2024-11-10 14:49:51 -08:00
sabaimran
170d959feb Handle offline messages differently, as they don't respond well to the structured messages 2024-11-09 19:52:46 -08:00
sabaimran
2c543bedd7 Add typing to the constructed messages listed 2024-11-09 19:40:27 -08:00
sabaimran
79b15e4594 Only add images when they're present and vision enabled 2024-11-09 19:37:30 -08:00
sabaimran
bd55028115 Fix randint import from random when creating filenames for tmp 2024-11-09 19:17:18 -08:00
sabaimran
92b6b3ef7b Add attached files to latest structured message in chat ml format 2024-11-09 19:17:00 -08:00
sabaimran
835fa80a4b Allow docx conversion in the chatFunction.ts 2024-11-09 18:51:00 -08:00
sabaimran
459318be13 And random suffixes to decreases any clash probability when writing tmp files to disc 2024-11-09 18:46:34 -08:00
sabaimran
dbf0c26247 Remove _summary_ description in function descriptions 2024-11-09 18:42:42 -08:00
sabaimran
e5ac076fc4 Move construct_chat_history method back to conversation.utils.py 2024-11-09 18:27:46 -08:00
sabaimran
bc95a99fb4 Make tracer the last input parameter for all the relevant chat helper methods 2024-11-09 18:22:46 -08:00
sabaimran
ceb29eae74 Add phone number verification and remove telemetry update call from place where authentication middleware isn't yet installed (in the middleware itself). 2024-11-09 12:25:36 -08:00
sabaimran
3badb27744 Remove stored uploaded files after they're processed. 2024-11-08 23:28:02 -08:00
sabaimran
78630603f4 Delete the fact checker application 2024-11-08 17:27:42 -08:00
sabaimran
807687a0ac Automatically generate titles for conversations from history 2024-11-08 16:02:34 -08:00
sabaimran
7159b0b735 Enforce limits on file size when converting to text 2024-11-08 15:27:28 -08:00
sabaimran
4695174149 Add support for file preview in the chat input area (before message sent) 2024-11-08 15:12:48 -08:00
sabaimran
ad46b0e718 Label pages when extract text from pdf, docs content. Fix scroll area in doc preview. 2024-11-08 14:53:20 -08:00
sabaimran
ee062d1c48 Fix parsing for PDFs via content indexing API 2024-11-07 18:17:29 -08:00
sabaimran
623a97a9ee Merge branch 'master' of github.com:khoj-ai/khoj into features/include-full-file-in-convo-with-filter 2024-11-07 17:18:23 -08:00
sabaimran
33498d876b Simplify the share chat page. Don't need it to maintain its own conversation history 
- When chatting on a shared page, fork and redirect to a new conversation page
 2024-11-07 17:14:11 -08:00
sabaimran
4b8be55958 Convert UUID to string when forking a conversation 2024-11-07 17:13:04 -08:00
sabaimran
9bbe27fe36 Set default value of attached files to empty list 2024-11-07 17:12:45 -08:00
sabaimran
3a51996f64 Process attached files in the chat history and add them to the chat message 2024-11-07 16:06:58 -08:00
sabaimran
a89160e2f7 Add support for converting an attached doc and chatting with it 
- Document is first converted in the chatinputarea, then sent to the chat component. From there, it's sent in the chat API body and then processed by the backend
- We couldn't directly use a UploadFile type in the backend API because we'd have to convert the api type to a multipart form. This would require other client side migrations without uniform benefit, which is why we do it in this two-phase process. This also gives us capacity to repurpose the moe generic interface down the road.
 2024-11-07 16:06:37 -08:00
sabaimran
e521853895 Remove unnecessary console.log statements 2024-11-07 16:03:31 -08:00
sabaimran
92c3b9c502 Add function to get an icon from a file type 2024-11-07 16:02:53 -08:00
sabaimran
140c67f6b5 Remove focus ring from the text area component 2024-11-07 16:02:02 -08:00
sabaimran
b8ed98530f Accept attached files in the chat API 
- weave through all subsequent subcalls to models, where relevant, and save to conversation log
 2024-11-07 16:01:48 -08:00
sabaimran
ecc81e06a7 Add separate methods for docx and pdf files to just convert files to raw text, before further processing 2024-11-07 16:01:08 -08:00
sabaimran
394035136d Add an api that gets a document, and converts it to just text 2024-11-07 16:00:10 -08:00
sabaimran
3b1e8462cd Include attach files in calls to extract questions 2024-11-07 15:59:15 -08:00
sabaimran
de73cbc610 Add support for relaying attached files through backend calls to models 2024-11-07 15:58:52 -08:00
Debanjum
4cad96ded6 Add Script to Evaluate Khoj on Google's FRAMES benchmark (#955) 
- Why
We need better, automated evals to measure performance shifts of Khoj
across prompt, model and capability changes.

Google's FRAMES benchmark evaluates multi-step retrieval and reasoning
capabilities of AI agents. It's a good starter benchmark to evaluate Khoj.

- Details
This PR adds an eval script to evaluate Khoj responses on the the FRAMES
benchmark prompts against the ground truth provided by it.

Script allows configuring sample size, batch size, sampling queries from the
eval dataset.

Gemini is used as an LLM Judge to auto grade Khoj responses vs ground truth 
data from the benchmark.
 2024-11-06 17:52:01 -08:00
Debanjum
8679294bed Remove need to set server chat settings from use openai proxies docs 
This was previously required, but now it's only usefuly for more
advanced settings, not typical for self-hosting users.

With recent updates, the user's selected chat model is used for both
Khoj's train of thought and response. This makes it easy to
switch your preferred chat model directly from the user settings
page and not have to update this in the admin panel as well.

Reflect these code changse in the docs, by removing the unnecessary
step for self-hosted users to create a server chat setting when using
an OpenAI proxy service like Ollama, LiteLLM etc.
 2024-11-05 17:10:53 -08:00
Debanjum
05a93fcbed v-align attach, send buttons with chat input text area on web app 
Otherwise, those buttons look off-center when images are attached to
the chat input area
 2024-11-05 17:10:53 -08:00
sabaimran
a0480d5f6c use fill weight for the toggle right (enabled state) for research mode 2024-11-04 22:01:09 -08:00
sabaimran
dc26da0a12 Add uploaded files in the conversation file filter for a new convo 2024-11-04 22:00:47 -08:00
Debanjum
b51ee644aa Fix escaping filename when normalizing in org node parser 2024-11-04 20:24:57 -08:00
Debanjum
5724d16a6f Fix passing images to anthropic chat models to extract questions 2024-11-04 20:24:57 -08:00
sabaimran
cf0bcec0e7 Revert SKIP_TESTS flag in offline chat director tests 2024-11-04 19:06:54 -08:00
sabaimran
1f372bf2b1 Update file summarization unit tests now that multiple files are allowed 2024-11-04 17:45:54 -08:00
sabaimran
7543360210 Merge branch 'master' of github.com:khoj-ai/khoj into features/include-full-file-in-convo-with-filter 2024-11-04 16:55:48 -08:00
sabaimran
b6145df3be Handle file retrieval when agent is None 2024-11-04 16:55:22 -08:00
sabaimran
3dc9139cee Add additional handling for when file_object comes back empty 2024-11-04 16:53:07 -08:00
sabaimran
a27b8d3e54 Remove summarize condition for only 1 file filter 2024-11-04 16:51:37 -08:00
sabaimran
362bdebd02 Add methods for reading full files by name and including context 
Now that models have much larger context windows, we can reasonably include full texts of certain files in the messages. Do this when an explicit file filter is set in a conversation. Do so in a separate user message in order to mitigate any confusion in the operation.

Pipe the relevant attached_files context through all methods calling into models.

We'll want to limit the file sizes for which this is used and provide more helpful UI indicators that this sort of behavior is taking place.
 2024-11-04 16:37:13 -08:00
sabaimran
e3ca52b7cb Use .get() to get text accompanying image url, instead of subindexing 2024-11-04 16:09:16 -08:00
sabaimran
1e89baca7b Deprecate the UserSearchModelConfig and remove all references 
- The server has moved to a model of standardization for the embeddings generation workflow. Remove references to the support for differentiated models.
- The migration script fo ra new model needs to be updated to accommodate full regeneration.
 2024-11-04 12:24:41 -08:00
Debanjum
1ccbf72752 Use logger instead of print to track eval 2024-11-04 00:40:26 -08:00
sabaimran
99c1d2831a Release Khoj version 1.28.3 2024-11-02 12:23:11 -07:00
sabaimran
075b4ecf15 Call subscription_to_state with sync_to_async wrapper when getting user subscription state 
- This is needed in case the renewal_date is not set and we need to reset it for the user
 2024-11-02 12:22:35 -07:00
sabaimran
ec44cbe1e7 Release Khoj version 1.28.2 2024-11-02 07:53:51 -07:00
Debanjum
791eb205f6 Run prompt batches in parallel for faster eval runs 2024-11-02 04:58:03 -07:00
Debanjum
96904e0769 Add script to evaluate khoj on Google's FRAMES benchmark 
Google's FRAMES benchmark evaluates multi-step retrieval and reasoning
capabilities of an agent.

The script uses Gemini as an LLM Judge to evaluate Khoj responses to
the FRAMES benchmark prompts against the ground truth provided by it.
 2024-11-02 04:57:42 -07:00
Debanjum
31b5fde163 Only enable prompt tracer if git python is installed 2024-11-02 02:07:02 -07:00
sabaimran
5b18dc96e0 Release Khoj version 1.28.1 2024-11-01 22:51:51 -07:00
sabaimran
8d1b1bc78e Move the git python dependency into top level dependencies 2024-11-01 22:51:00 -07:00
Debanjum
e85dd59295 Release Khoj version 1.28.0 2024-11-01 19:06:59 -07:00
Debanjum
1f79a10541 Fix link to code execution feature in docs 2024-11-01 18:22:21 -07:00
Debanjum
cff8e02b60 Research Mode [Part 2]: Improve Prompts, Edit Chat Messages. Set LLM Seed for Reproducibility (#954) 
- Improve chat actors and their prompts for research mode.
- Add documentation to enable the code tool when self-hosting Khoj
- Edit Chat Messages
  - Store Turn Id in each chat message. 
  - Expose API to delete chat message.
  - Expose delete chat message button to turn delete chat message from web app
- Set LLM Generation Seed for Reproducible Debugging and Testing
  - Setting seed for LLM generation is supported by Llama.cpp and OpenAI models. 
    This can (somewhat) restrain LLM output
  - Getting fixed responses for fixed inputs helps test, debug longer reasoning chains like used in advanced reasoning
 2024-11-01 18:16:42 -07:00
Debanjum
14e453039d Add prompt tracing, agent personality to infer webpage urls chat actor 2024-11-01 18:12:50 -07:00
Debanjum
ab321dc518 Expect query before tool in response to give think space in research prompt 2024-11-01 17:51:41 -07:00
Debanjum
1a83bbcc94 Clean API chat router. Move FeedbackData response type to router helper 2024-11-01 17:51:41 -07:00
sabaimran
e6eb87bbb5 Merge branch 'improve-debug-reasoning-and-other-misc-fixes' of github.com:khoj-ai/khoj into improve-debug-reasoning-and-other-misc-fixes 2024-11-01 16:48:39 -07:00
sabaimran
a213b593e8 Limit the number of urls the webscraper can extract for scraping 2024-11-01 16:48:36 -07:00
sabaimran
327fcb8f62 create defiltered query after conversation command is extracted 2024-11-01 16:48:03 -07:00
sabaimran
b79a9ec36d Clarify description of the code evaluation environment: not for document creation 2024-11-01 16:47:27 -07:00
Debanjum
9c7b36dc69 Use standard per minute rate limits across user types 2024-11-01 16:16:06 -07:00
Debanjum
ac21b10dd5 Simplify logic to get default search model. Remove unused import 2024-11-01 15:14:00 -07:00
sabaimran
2b35790165 Merge branch 'master' of github.com:khoj-ai/khoj into improve-debug-reasoning-and-other-misc-fixes 2024-11-01 14:51:26 -07:00
Debanjum
22f3ed3f5d Research Mode: Give Khoj the ability to perform more advanced reasoning (#952) 
## Overview
Khoj can now go into research mode and use a python code interpreter. These are experimental features that are being released early for feedback and testing.

- Research mode allows Khoj to dynamically select the tools it needs to best answer the question. It is also allowed more iterations to get to a satisfactory answer. Its more dynamic train of thought is shown to improve visibility into its thinking.
- Adding ability for Khoj to use a python code interpreter is an adjacent capability. It can help Khoj do some data analysis and generate charts for you. A sandboxed python to run code is provided using [cohere-terrarium](https://github.com/cohere-ai/cohere-terrarium?tab=readme-ov-file), [pyodide](https://pyodide.org/).

## Analysis
Research mode (significantly?) improves Khoj's information retrieval for more complex queries requiring multi-step lookups but takes longer to run. It can research for longer, requiring less back-n-forth with the user to find an answer.

Research mode gives most gains when used with more advanced chat models (like o1, 4o, new claude sonnet and gemini-pro-002). Smaller models improve their response quality but tend to get into repetitive loops more often. 

## Next Steps
- Get community feedback on research mode. What works, what fails, what is confusing, what'd be cool to have.
- Tune Khoj's capabilities for longer autonomous runs and to generalize across a larger range of model sizes

## Miscellaneous Improvements
- Khoj's train of thought is saved and shown for all messages, not just the latest one
- Render charts generated by Khoj and code running using the code tool on the web app
- Align chat input color to currently selected agent color
 2024-11-01 14:46:29 -07:00
sabaimran
baa939f4ce When running code, strip any code delimiters. Disable application json type specification in Gemini request. 2024-11-01 13:47:39 -07:00
sabaimran
8fd2fe162f Determine if research mode is enabled by checking the conversation commands and 'linting' them in the selection phase 2024-11-01 13:12:34 -07:00
sabaimran
cead1598b9 Don't reset research mode after completing research execution 2024-11-01 13:00:11 -07:00
Debanjum
c1c779a7ef Do not yaml format raw code results in context for LLM. It's confusing 2024-11-01 12:45:26 -07:00
sabaimran
b3dad1f393 Standardize rate limits to 1/6 ratio 2024-11-01 12:21:09 -07:00
sabaimran
23a49b6b95 Add documentation for python code execution capability 2024-11-01 12:14:33 -07:00
Debanjum
cd75151431 Do not allow auto selecting research mode as tool for now. 
You are required to manually turning it on. This takes longer and
should be a high intent activity initiated by user
 2024-11-01 12:07:52 -07:00
Debanjum
0b0cfb35e6 Simplify in research mode check in api_chat. 
- Dedent code for readability
- Use better name for in research mode check
- Continue to remove inferred summarize command when multiple files in
  file filter even when not in research mode
- Continue to show select information source train of thought.
  It was removed by mistake earlier
 2024-11-01 12:07:08 -07:00
sabaimran
ffa7f95559 Add template for a code sandbox to the docker-compose configuration 2024-11-01 11:50:58 -07:00
Debanjum
73750ef286 Merge branch 'master' into features/advanced-reasoning 2024-11-01 11:42:01 -07:00
sabaimran
1fc280db35 Handle case where infer_webpage_url returns no valid urls 2024-11-01 11:41:32 -07:00
Debanjum
1c920273dd Add Prompt Tracer to Visualize, Analyze and Debug Khoj's Train of Thought (#951) 
## Overview
Use git to capture prompt traces of khoj's train of thought. View, analyze and debug them using your favorite git client (e.g vscode, magit).

- Each commit captures an interaction with an LLM
  The commit writes the query, response and system message each to a separate file in the repo.
  The commit message captures the chat model, Khoj version and other metadata
- Each conversation turn can have multiple interactions with an LLM (e.g Khoj's train of thought)
- Each new conversation turn forks from and merges back into its conversation branch
- Each new conversation branches from the user branch
- Each new user branches from root commit on the main branch

## Usage
1. Set `KHOJ_DEBUG=true` or start khoj in very verbose mode with `khoj -vv` to turn on prompt tracing
2. Chat with Khoj as usual 
3. Open the promptrace git repo to view the generated prompt traces using your favorite git porcelain. 
   The Khoj prompt trace git repo is created at `/tmp/khoj_promptrace` by default. You can configure the prompt trace directory by setting the `PROMPTRACE_DIR`environment variable.

## Implementation
- Add utility functions to capture prompt traces using git (via `gitpython`)
- Make each model provider in Khoj commit their LLM interactions with promptrace
- Weave chat metadata from chat API through all chat actors and commit it to the prompt trace
 2024-11-01 11:33:54 -07:00
sabaimran
33d36ee58c Add experimental notice to research mode tooltip 2024-11-01 11:00:27 -07:00
sabaimran
0145b2a366 Set usage limits on the research mode 2024-11-01 10:29:33 -07:00
sabaimran
3ea94ac972 Only include inferred-queries in chat history when present 2024-10-31 22:01:41 -07:00
sabaimran
149cbe1019 Use bottom anchor for the commandbar popover 2024-10-31 20:40:38 -07:00
sabaimran
21858acccc Remove conversation command always in query, filter out inferred queries that were not with selected tool when going through tool selection iterations 2024-10-31 20:27:38 -07:00
sabaimran
19241805ee Merge branch 'master' of github.com:khoj-ai/khoj into improve-debug-reasoning-and-other-misc-fixes 2024-10-31 18:20:23 -07:00
Debanjum
302bd51d17 Improve online chat actor prompt for research and normal mode 
- Match the online query generator prompt to match the formatting of
  extract questions
- Separate iteration results by newline
- Improve webpage and online tool descriptions
 2024-10-31 18:17:12 -07:00
Debanjum
52163fe299 Improve research planner prompt to reduce looping 2024-10-31 18:17:01 -07:00
sabaimran
7ebf999688 Merge branch 'master' of github.com:khoj-ai/khoj into features/advanced-reasoning 2024-10-31 18:15:13 -07:00
sabaimran
159ea44883 Remove frame references in the diagramming prompts 2024-10-31 18:14:51 -07:00
Debanjum
89597aefe9 Json dump contents in prompt tracer to make structure discernable 2024-10-31 18:08:42 -07:00
Debanjum
5b15176e20 Only add /research prefix in research mode if not already in user query 2024-10-31 18:08:42 -07:00
sabaimran
559601dd0a Do not exit if/else loop in research loop when notes not found 2024-10-31 13:51:10 -07:00
sabaimran
a13760640c Only show trash can when turnId is present 2024-10-31 13:19:16 -07:00
sabaimran
8d1ecb9bd8 Add optional brew steps for docker install 2024-10-31 12:41:53 -07:00
Debanjum
adca6cbe9d Merge branch 'master' into add-prompt-tracer-for-observability 2024-10-31 02:28:34 -07:00
Debanjum
e17dc9f7b5 Put train of thought ui before Khoj response on web app 2024-10-31 02:24:53 -07:00
Debanjum
e8e6ead39f Fix deleting new messages generated after conversation load 2024-10-30 20:56:38 -07:00
Debanjum
cb90abc660 Resolve train of thought component needs unique key id error on web app 2024-10-30 14:00:21 -07:00
Debanjum
ca5a6831b6 Add ability to delete messages from the web app 2024-10-30 14:00:21 -07:00
Debanjum
ba15686682 Store turn id with each chat message. Expose API to delete chat turn 
Each chat turn is a user query, khoj response message pair
 2024-10-30 14:00:21 -07:00
Debanjum
f64f5b3b6e Handle add/delete file filter operation on non-existent conversation 2024-10-30 14:00:21 -07:00
Debanjum
b3a63017b5 Support setting seed for reproducible LLM response generation 
Anthropic models do not support seed. But offline, gemini and openai
models do. Use these to debug and test Khoj via KHOJ_LLM_SEED env var
 2024-10-30 14:00:21 -07:00
Debanjum
d44e68ba01 Improve handling embedding model config from admin interface 
- Allow server to start if loading embedding model fails with an error.
  This allows fixing the embedding model config via admin panel.

  Previously server failed to start if embedding model was configured
  incorrectly. This prevented fixing the model config via admin panel.

- Convert boolean string in config json to actual booleans when passed
  via admin panel as json before passing to model, query configs

- Only create default model if no search model configured by admin.
  Return first created search model if its been configured by admin.
 2024-10-30 14:00:21 -07:00
Debanjum
358a6ce95d Defer turning cursor color to selected agents color for later 
Capability exists but idea needs to be investigated further
 2024-10-30 14:00:21 -07:00
Debanjum
2ac840e3f2 Make cursor in chat input take on selected agent color 2024-10-30 14:00:21 -07:00
Debanjum
1448b8b3fc Use 3rd person for user in research prompt to reduce person confusion 
Models were getting a bit confused about who is search for who's
information. Using third person to explicitly call out on who's behalf
these searches are running seems to perform better across
models (gemini's, gpt etc.), even if the role of the message is user.
 2024-10-30 13:49:48 -07:00
Debanjum
b8c6989677 Separate example from actual question in extract question prompt 2024-10-30 13:49:48 -07:00
Debanjum
86ffd7a7a2 Handle \n, dedupe json cleaning into single function for reusability 
Use placeholder for newline in json object values until json parsed
and values extracted. This is useful when research mode models outputs
multi-line codeblocks in queries etc.
 2024-10-30 13:49:48 -07:00
Debanjum
83ca820abe Encourage Anthropic models to output json object using { prefill 
Anthropic API doesn't have ability to enforce response with valid json
object, unlike all the other model types.

While the model will usually adhere to json output instructions.

This step is meant to more strongly encourage it to just output json
object when response_type of json_object is requested.
 2024-10-30 13:49:48 -07:00
Debanjum
dc8e89b5de Pass tool AIs iteration history as chat history for better context 
Separate conversation history with user from the conversation history
between the tool AIs and the researcher AI.

Tools AIs don't need top level conversation history, that context is
meant for the researcher AI.

The invoked tool AIs need previous attempts at using the tool in this
research runs iteration history to better tune their next run.

Or at least that is the hypothesis to break the models looping.
 2024-10-30 13:49:48 -07:00
Debanjum
d865994062 Rename code tool arg previous_iteration_history' to context' 2024-10-30 13:49:48 -07:00
Debanjum
06aeca2670 Make researcher, docs search AIs ask more diverse retrieval questions 
Models weren't generating a diverse enough set of questions. They'd do
minor variations on the original query. What is required is asking
queries from a bunch of different lenses to retrieve the requisite
information.

This prompt updates shows the AIs the breadth of questions to by
example and instruction. Seem like performance improved based on vibes
 2024-10-30 13:49:48 -07:00
Debanjum
01881dc7a2 Revert "Make extract question prompt in 1st person wrt user as its a user message" 
This reverts commit 6d3602798aa1b95a30c557576fd4f93ddef2ae76.
 2024-10-30 13:49:48 -07:00
Debanjum
3e695df198 Make extract question prompt in 1st person wrt user as its a user message 
Divide Example from Actual chat history section in prompt
 2024-10-30 13:49:48 -07:00
Debanjum
a3751d6a04 Make extract relevant information system prompt work for any document 
Previously it was too strongly tuned for extracting information from
only webpages. This shouldn't be necessary
 2024-10-30 13:49:48 -07:00
Debanjum
a39e747d07 Improve passing user name in pick next research tool prompt 2024-10-30 13:49:48 -07:00
Debanjum
deff512baa Improve research mode prompts to reduce looping, increase webpage reads 2024-10-30 13:49:48 -07:00
Debanjum
d3184ae39a Simplify storing and displaying document results in research mode 
- Mention count of notes and files disovered
- Store query associated with each compiled reference retrieved for
  easier referencing
 2024-10-30 13:49:48 -07:00
Debanjum
8bd94bf855 Do not use a message branch if no msg id provided to prompt tracer 2024-10-30 13:49:48 -07:00
sabaimran
b63fbc5345 Add a simple badget to the dropdown menu that shows subscription status 2024-10-30 13:00:16 -07:00
sabaimran
82f3d79064 Merge branch 'master' of github.com:khoj-ai/khoj into features/advanced-reasoning 2024-10-30 11:32:10 -07:00
sabaimran
2b2564257e Handle subscription case where it's set to trial, but renewal_date is not set. set the renewal_date for LENGTH_OF_FREE_TRIAL days from subscription creation. 2024-10-30 11:05:31 -07:00
Debanjum
9935d4db0b Do not use a message branch if no msg id provided to prompt tracer 2024-10-28 17:50:27 -07:00
Debanjum
d184498038 Pass context in separate message from user query to research chat actor 2024-10-28 15:37:28 -07:00
Debanjum
d75ce4a9e3 Format online, notes, code context with YAML to be legibile for LLM 2024-10-28 15:37:28 -07:00
sabaimran
5bea0c705b Use break-words in the train of thought for better formatting 2024-10-28 15:36:06 -07:00
sabaimran
1f1b182461 Automatically carry over research mode from home page to chat 
- Improve mobile friendliness with new research mode toggle, since chat input area is now taking up more space
- Remove clunky title from the suggestion card
- Fix fk lookup error for agent.creator
 2024-10-28 15:29:24 -07:00
sabaimran
ebaed53069 Merge branch 'master' of github.com:khoj-ai/khoj into features/advanced-reasoning 2024-10-28 12:39:00 -07:00
sabaimran
889dbd738a Add keyword diagram to diagram output mode description 2024-10-28 12:20:46 -07:00
Debanjum
50ffd7f199 Merge branch 'master' into features/advanced-reasoning 2024-10-28 04:10:59 -07:00
Debanjum
a5d0ca6e1c Use selected agent color to theme the chat input area on home page 2024-10-28 03:47:40 -07:00
Debanjum
aad7528d1b Render slash commands popup below chat input text area on home page 2024-10-28 02:06:04 -07:00
Debanjum
3e17ab438a Separate notes, online context from user message sent to chat models (#950) 
Overview
---
- Put context into separate user message before sending to chat model.
  This should improve model response quality and truncation logic in code
- Pass online context from chat history to chat model for response.
  This should improve response speed when previous online context can be reused
- Improve format of notes, online context passed to chat models in prompt.
  This should improve model response quality

Details
---
The document, online search context are now passed as separate user
messages to chat model, instead of being added to the final user message.

This will improve
- Models ability to differentiate data from user query.
  That should improve response quality and reduce prompt injection
  probability
- Make truncation logic simpler and more robust
  When context window hit, can simply pop messages to auto truncate
  context in order of context, user, assistant message for each
  conversation turn in history until reach current user query

  The complex, brittle logic to extract user query from context in
  last user message isn't required.
 2024-10-28 02:03:18 -07:00
Debanjum
8ddd70f3a9 Put context into separate message before sending to offline chat model 
Align context passed to offline chat model with other chat models

- Pass context in separate message for better separation between user
  query and the shared context
- Pass filename in context
- Add online results for webpage conversation command
 2024-10-28 00:22:21 -07:00
Debanjum
ee0789eb3d Mark context messages with user role as context role isn't being used 
Context role was added to allow change message truncation order based
on context role as well.

Revert it for now since currently this is not currently being done.
 2024-10-28 00:04:14 -07:00
Debanjum
4e39088f5b Make agent name in home page carousel not text wrap on mobile 2024-10-27 23:03:53 -07:00
Debanjum
94074b7007 Focus chat input on toggle research mode. v-align it with send button 2024-10-27 22:54:55 -07:00
sabaimran
a691ce4aa6 Batch entries into smaller groups to process 2024-10-27 20:43:41 -07:00
sabaimran
2924909692 Add a research mode toggle to the chat input area 2024-10-27 16:37:40 -07:00
sabaimran
68499e253b Auto-collapse train of thought, show after chat response in history 2024-10-27 15:48:13 -07:00
sabaimran
101ea6efb1 Add research mode as a slash command, remove from default path 2024-10-27 15:47:44 -07:00
sabaimran
0bd78791ca Let user exit from command mode with esc, click out, etc. 2024-10-27 15:01:49 -07:00
sabaimran
a121d67b10 Persist the train of thought in the conversation history 2024-10-26 23:46:15 -07:00
sabaimran
9e8ac7f89e Fix input/output mismatches in the /summarize command 2024-10-26 16:37:58 -07:00
sabaimran
e4285941d1 Use the advanced chat model if the user is subscribed 2024-10-26 16:00:54 -07:00
sabaimran
33e48aa27e Merge branch 'add-prompt-tracer-for-observability' of github.com:khoj-ai/khoj into features/advanced-reasoning 2024-10-26 14:09:00 -07:00
sabaimran
fd71a4b086 Add better exception handling in the prompt trace logic, use default value from parameters 2024-10-26 14:08:00 -07:00
Debanjum
3e5b5ec122 Encourage model to read webpages more often after online search 
Previously model would rarely read webpages after webpage search. Need
the model to webpages more regularly for deeper research and to stop
getting stuck in repetitive online search loops
 2024-10-26 10:49:09 -07:00
Debanjum
bf96d81943 Format online results as YAML to pass it in more readable form to model 
Previous passing of online results as json dump in prompts was less
readable for humans, and I'm guessing less readable for
models (trained on human data) as well?
 2024-10-26 10:49:09 -07:00
Debanjum
3e97ebf0c7 Unescape special characters in prompt traces for better readability 2024-10-26 10:49:09 -07:00
Debanjum
8af9dc3ee1 Unescape special characters in prompt traces for better readability 2024-10-26 10:45:42 -07:00
Debanjum Singh Solanky
0f3927e810 Send gathered references to client after code results calculated 2024-10-26 05:59:10 -07:00
Debanjum Singh Solanky
f04f871a72 Merge branch 'add-prompt-tracer-for-observability' of github.com:khoj-ai/khoj into features/advanced-reasoning 
- Start from this branches src/khoj/routers/api_chat.py
    Add tracer to all old and new chat actors that don't have it set
    when they are called.
  - Update the new chat actors like apick next tool etc to use tracer too
 2024-10-26 05:56:13 -07:00
Debanjum Singh Solanky
ddc6ccde2d Merge branch 'master' into features/advanced-reasoning 
- Conflicts:
  Combine both sides of the conflict in all 3 files below
  - src/khoj/processor/conversation/utils.py
  - src/khoj/routers/helpers.py
  - src/khoj/utils/helpers.py
 2024-10-26 05:15:51 -07:00
Debanjum Singh Solanky
ea0712424b Commit conversation traces using user, chat, message branch hierarchy 
- Message train of thought forks and merges from its conversation branch
- Conversation branches from user branch
- User branches from root commit on the main branch

- Weave chat tracer metadata from api endpoint through all chat actors
  and commit it to the prompt trace
 2024-10-26 05:08:47 -07:00
Debanjum Singh Solanky
a3022b7556 Allow Offline Chat model calling functions to save conversation traces 2024-10-26 05:08:47 -07:00
Debanjum Singh Solanky
eb6424f14d Allow Anthropic API calling functions to save conversation traces 2024-10-26 05:08:47 -07:00
Debanjum Singh Solanky
6fcd6a5659 Allow Gemini API calling functions to save conversation traces 2024-10-26 05:08:47 -07:00
Debanjum Singh Solanky
384f394336 Allow OpenAI API calling functions to save conversation traces 2024-10-26 04:59:21 -07:00
Debanjum Singh Solanky
10c8fd3b2a Save conversation traces to git for visualization 2024-10-26 04:59:19 -07:00
sabaimran
7e0a692d16 Release Khoj version 1.27.1 2024-10-25 15:23:07 -07:00
sabaimran
b257fa1884 Add a None check before doing a DT comparison when getting subscription type 2024-10-25 15:22:48 -07:00
sabaimran
0f6f282c30 Release Khoj version 1.27.0 2024-10-25 14:11:14 -07:00
sabaimran
479e156168 Add to the ConversationCommand.Image description to LLM 2024-10-25 09:14:32 -07:00
sabaimran
a11b5293fb Add uploaded images to research mode, code slash command, include code references 2024-10-24 23:56:24 -07:00
sabaimran
5acf40c440 Clean up summarization code paths 
Use assumption of summarization response being a str
 2024-10-24 23:56:24 -07:00
sabaimran
12b32a3d04 Resolve merge conflicts 2024-10-24 23:43:55 -07:00
Debanjum
adee5a3e20 Give Vision to Anthropic models in Khoj (#948) 
### Major
- Give Vision to Anthropic models in Khoj

### Minor
- Reuse logic to format messages for chat with anthropic models
- Make the get image from url function more versatile and reusable
- Encourage output mode chat actor to output only json and nothing else
 2024-10-24 18:02:38 -07:00
Debanjum Singh Solanky
01d740debd Return typed image from image_with_url function for readability 2024-10-24 17:58:46 -07:00
Debanjum Singh Solanky
37317e321d Dedupe user location passed in image, diagram generation prompts 2024-10-24 01:03:29 -07:00
Debanjum Singh Solanky
2a32836d1a Log more descriptive error when image gen fails with Replicate 2024-10-24 01:03:29 -07:00
sabaimran
30f9225021 Merge branch 'master' of github.com:khoj-ai/khoj into features/advanced-reasoning 2024-10-23 19:15:51 -07:00
sabaimran
5120597d4e Remove user customized search model (#946) 
- Use a single standard search model across the server. There's diminishing benefits for having multiple user-customizable search models. 
- We may want to add server-level customization for specific tasks
- Store the search model used to generate a given entry on the `Entry` object
- Remove user-facing APIs and view
- Add a management command for migrating the default search model on the server

In a future PR (after running the migration), we'll also remove the `UserSearchModelConfig`
 2024-10-23 17:38:37 -07:00
Debanjum Singh Solanky
8d588e0765 Encourage output mode chat actor to output only json and nothing else 
Latest claude model wanted to say more than just give the json output.
The updated prompt encourages the model to ouput just json. This is
similar to what is already being done for other prompts
 2024-10-23 17:19:21 -07:00
Debanjum Singh Solanky
abad5348a0 Give Vision to Anthropic models in Khoj 2024-10-23 17:19:21 -07:00
Debanjum Singh Solanky
6fd50a5956 Reuse logic to format messages for chat with anthropic models 2024-10-23 17:19:21 -07:00
Debanjum Singh Solanky
82eac5a043 Make the get image from url function more versatile and reusable 
It was previously added under the google utils. Now it can be used by
other conversation processors as well.

The updated function
- can get both base64 encoded and PIL formatted images from url
- will return the media type of the image as well in response
 2024-10-23 17:19:20 -07:00
sabaimran
f3ce47b445 Create explicit flow to enable the free trial (#944) 
* Create explicit flow to enable the free trial

The current design is confusing. It obfuscates the fact that the user is on a free trial. This design will make the opt-in explicit and more intuitive.

* Use the Subscription Type enum instead of hardcoded strings everywhere

* Use length of free trial in the frontend code as well
 2024-10-23 15:29:23 -07:00
Debanjum Singh Solanky
bc059eeb0b Merge branch 'master' into put-retrieved-context-in-separate-chatml-message 2024-10-23 12:55:18 -07:00
Debanjum Singh Solanky
3b978b9b67 Fix chat history construction when generating chatml msgs with context 2024-10-23 12:55:12 -07:00
sabaimran
c5e91c346a Fix Docker desktop link for Linux 2024-10-23 11:24:54 -07:00
Debanjum Singh Solanky
9f2c02d9f7 Chat with the default agent by default from web app home 
Had temporarily updated the default selected agent to last used.
Revert for now as
1. The previous logic was buggy. It didn't select the default agent
   even when the last used agent was the default agent. Which would
   require more work.
2. It maybe too early anyway to set the default agent to last used.
 2024-10-23 03:43:57 -07:00
Debanjum Singh Solanky
218946edda Fix copying message with user images on web app 
Adding div elements to message to render degraded text copied to
clipboard for messages with user uploaded images.

This change fixes that by separating message to render from message
for clipboard. It ensures differently formatted forms of the user
images are added to the two to allow proper rendering while still
having decently formatted text copied to clipboard
 2024-10-23 03:41:25 -07:00
Debanjum Singh Solanky
7d9a06c8ab Merge branch 'master' into put-retrieved-context-in-separate-chatml-message 2024-10-23 00:13:38 -07:00
sabaimran
7c29af9745 Add link to self-hosted admin page and add docs for building front-end assets. Close #901 2024-10-22 22:42:27 -07:00
Debanjum Singh Solanky
2a50694089 Allow typing multi-line queries from a phone with Enter key 
Add newline instead of sending message when hit Enter key on mobile
displays. As on phones shift key doesn't exist and send button is easily
clickable.

Limit hitting Enter key to send message to computers = larger display
= expected to have full fledged keyboards.
 2024-10-22 21:20:22 -07:00
Debanjum Singh Solanky
a134cd835c Focus on chat input area to enter text after file uploads on web app 2024-10-22 21:19:17 -07:00
Debanjum
c81e708833 Show all agents, smart sorted, in carousel on home screen of web app (#943) 
## Overview
Allow quickly selecting, switching agents from agents pane on home page of web app

## Details
- Show all agents in carousel on home screen agent pane of web app
- Smart Sort
  1. Pin default agent as first for ease of access
  2. Show used agents by MRU for ease of access
  3. Shuffle unused agents for discoverability
- Select most recently used agent to chat with by default
- Push smart sort logic down to API
  - Common logic can be reused across clients
  - Agent sort was previously done in web app
- Focus on chat input on agent select
- Double click agent on home page to open edit agent card on agents page
 2024-10-22 21:18:17 -07:00
Debanjum Singh Solanky
750fbce0c2 Merge branch 'master' into improve-agent-pane-on-home-screen 2024-10-22 20:05:29 -07:00
Debanjum Singh Solanky
3be505db48 Only show type of error when image generation fails to clients 
Rather than showing raw error message from the underlying service as it
could contain sensitive information
 2024-10-22 20:03:20 -07:00
Debanjum
c6f3253ebd Chat with Multiple Images. Support Vision with Gemini (#942) 
## Overview
- Add vision support for Gemini models in Khoj
- Allow sharing multiple images as part of user query from the web app
- Handle multiple images shared in query to chat API
 2024-10-22 19:59:18 -07:00
Debanjum Singh Solanky
b3fff43542 Sanitize user attached images. Constrain chat input width on home page 
Set max combined images size to 20mb to allow multiple photos to be shared
 2024-10-22 19:42:40 -07:00
Debanjum Singh Solanky
6c393800cc Merge branch 'master' into multi-image-chat-and-vision-for-gemini 2024-10-22 18:38:49 -07:00
Debanjum Singh Solanky
91bbd19333 Close the agent detail hover card when scroll on agent pane 2024-10-22 18:03:17 -07:00
Debanjum Singh Solanky
110c67f083 Improve agent pill, detail card styling. Handle null chatInputRef 
- Remove border from agent detail hover card on home page
- Do not wrap long agent names in agent pills on home page
- Handle scenario where chatInputRef is null
 2024-10-22 18:03:17 -07:00
Debanjum Singh Solanky
aca8bef024 Only use recent chat sessions for agent MRU. Handle null agent chats 2024-10-22 17:46:45 -07:00
sabaimran
0dad4212fa Generate dynamic diagrams (via Excalidraw) (#940) 
Add support for generating dynamic diagrams in flow with Excalidraw (https://github.com/excalidraw/excalidraw). This happens in three steps:
1. Default information collection & intent determination step.
2. Improving the overall guidance of the prompt for generating a JSON, Excalidraw-compatible declaration.
3. Generation of the diagram to output to the final UI.

Add support in the web UI.
 2024-10-22 16:13:46 -07:00
sabaimran
1e993d561b Release Khoj version 1.26.4 2024-10-22 13:50:08 -07:00
Debanjum Singh Solanky
e8fb79a369 Rate limit the count and total size of images shared via API 2024-10-22 04:37:54 -07:00
Debanjum Singh Solanky
39a613d3bc Fix up openai chat actor tests 2024-10-22 03:09:36 -07:00
Debanjum Singh Solanky
0847fb0102 Pass online context from chat history to chat model for response 
Previously only notes context from chat history was included.
This change includes online context from chat history for model to use
for response generation.

This can reduce need for online lookups by reusing previous online
context for faster responses. But will increase overall response time
when not reusing past online context, as faster context buildup per
conversation.

Unsure if inclusion of context is preferrable. If not, both notes and
online context should be removed.
 2024-10-22 03:09:36 -07:00
Debanjum Singh Solanky
0c52a1169a Put context into separate user message before sending to chat model 
The document, online search context are now passed as separate user
messages to chat model, instead of being added to the final user message.

This will improve

- Models ability to differentiate data from user query.
  That should improve response quality and reduce prompt injection
  probability

- Make truncation logic simpler and more robust
  When context window hit, can simply pop messages to auto truncate
  context in order of context, user, assistant message for each
  conversation turn in history until reach current user query

  The complex, brittle logic to extract user query from context in
  last user message isn't required.

Marking the context message with assistant role doesn't translate well
across chat models. E.g
- Gemini can't handle consecutive messages by role = model well
- Claude will merge consecutive messages by same role. In current
  message ordering the context message will result get merged into the
  previous assistant response. And if move context message after user
  query. The truncation logic will have to hop and skip while doing
  deletions
- GPT seems to handle consecutive roles of any type fine

Using context role = user generalizes better across chat models for
now and aligns with previous behavior.
 2024-10-22 03:09:36 -07:00
Debanjum Singh Solanky
7ac241b766 Improve format of notes, online context passed to chat models in prompt 
Improve separation of note snippets and show its origin file in notes
prompt to have more readable, contextualized text shared with model.

Previously the references dict was being directly passed as a string.
The documents don't look well formatted and are less intelligible.

- Passing file path along with notes snippets will help contextualize
  the notes better.
- Better formatting should help with making notes more readable by the
  chat model.
 2024-10-22 03:09:36 -07:00
sabaimran
892040972f Replace user_id with server_id in telemetry 2024-10-21 20:47:52 -07:00
sabaimran
db959a504d Fix the version of pymupdf to avert build errors 2024-10-21 12:56:51 -07:00
sabaimran
21e69b506d Release Khoj version 1.26.3 2024-10-21 08:19:05 -07:00
Debanjum Singh Solanky
9b554feb91 Show agent details card on hover on agent pill on web app home page 
- Double click on agent to open edit agent card
- Focus on chat input pane when agent selected/clicked
  for quick, smooth agent switch and message flow
- Hover on agent to see agent detail card on non-mobile displays
  - Use debounce to only show when hover on card for a bit
 2024-10-21 00:08:01 -07:00
Debanjum Singh Solanky
220ff1df62 Set chatInputArea forward ref from parent components for control 2024-10-21 00:02:48 -07:00
Debanjum Singh Solanky
54b92eaf73 Extract isUserSubscribed check from Agents page to make it resusable 2024-10-20 23:31:48 -07:00
Debanjum Singh Solanky
bdbe8f003e Move agent details and edit card out into reusable components on web app 2024-10-20 23:31:47 -07:00
sabaimran
ad197be70c Fix PDFs unit test, skip OCR 2024-10-20 22:25:41 -07:00
sabaimran
59fec37943 Improve agents management, and limit agents view to private and official agents 
- Default to None for the input_tools and output_modes so that they can be managed in the admin panel
- Hold off on showing off all Public Agents until we have a better experience for user profiles etc.
 2024-10-20 22:24:51 -07:00
sabaimran
a979457442 Add unit tests for agents 
- Add permutations of testing for with, without knowledge base. Private, public, different users.
 2024-10-20 20:04:50 -07:00
Debanjum Singh Solanky
5fca41cc29 Show agents sorted by mru, Select mru agent by default on web app 
Have get agents API return agents ordered intelligently
- Put the default agent first
- Sort used agents by most recently chatted with agent for ease of access
- Randomly shuffle the remaining unused agents for discoverability
 2024-10-20 15:21:25 -07:00
Debanjum Singh Solanky
a6bfdbdbfe Show all agents in carousel on home screen agent pane of web app 
This change wraps the agent pane in a scroll area with all agents shown.
It allows selecting an agent to chat with directly from the home
screen without breaking flow and having to jump to the agents page.

The previous flow was not convenient to quickly and consistently start
chat with one of your standard agents.

This was because a random subet of agents were shown on the home page.
To start chat with an agent not shown on home screen load you had to
open the agents page and initiate the conversation from there.
 2024-10-20 15:21:25 -07:00
Debanjum Singh Solanky
7646ac6779 Style user attached images as carousel on chat input area of web app 2024-10-20 00:40:08 -07:00
sabaimran
5d5bea6a5f Ensure images are reset after messages processed 2024-10-19 22:02:06 -07:00
sabaimran
1ad6e1749f Move window redirect to after relevant data is dropped in localStorage on the homage page 
One limitation of this methodology is that localStorage has a limit in how much data it can take. Should add more graceful error handling here as well.
 2024-10-19 20:36:13 -07:00
sabaimran
cb6b3ec1e9 Improve mode description given to LLM when determining how to respond. 
Currently experiencing difficulty instruction following when an image is shared. It's more likely to try and output an image. Update to make a clearer distinction.
 2024-10-19 20:35:32 -07:00
sabaimran
545259e308 Remove unused icons in chatInputArea 2024-10-19 16:54:21 -07:00
Debanjum Singh Solanky
3cc1426edf Style user attached images with fixed height, in a single row on web app 2024-10-19 16:48:36 -07:00
Debanjum Singh Solanky
58a331227d Display the attached images inside the chat input area on the web app 
- Put the attached images display div inside the same parent div as
  the text area
- Keep the attachment, microphone/send message buttons aligned with
  the text area. So the attached images just show up at the top of the
  text area but everything else stays at the same horizontal height as
  before.

- This improves the UX by
  - Ensuring that the attached images do not obscure the agents pane
    above the chat input area
  - The attached images visually look like they are inside the actual
    input area, rather than floating above it. So the visual aligns
    with the semantics
 2024-10-19 16:29:45 -07:00
Debanjum Singh Solanky
3e39fac455 Add vision support for Gemini models in Khoj 2024-10-19 15:47:03 -07:00
Debanjum Singh Solanky
0d6a54c10f Allow sharing multiple images as part of user query from the web app 
Previously the web app only expected a single image to be shared by
the user as part of their query.

This change allows sharing multiple images from the web app.

Closes #921
 2024-10-19 15:47:03 -07:00
Debanjum Singh Solanky
e2abc1a257 Handle multiple images shared in query to chat API 
Previously Khoj could respond to a single shared image at a time.

This changes updates the chat API to accept multiple images shared by
the user and send it to the appropriate chat actors including the
openai response generation chat actor for getting an image aware
response
 2024-10-19 14:53:33 -07:00
Debanjum Singh Solanky
c6c48cfc18 Fix arg to generate_summary_from_file and type of this_iteration 2024-10-17 13:38:48 -07:00
Debanjum Singh Solanky
feb6d65ef8 Merge branch 'master' into features/advanced-reasoning 2024-10-15 09:37:56 -07:00
Debanjum Singh Solanky
336c6c3689 Show tool to use decision for next iteration in train of thought 2024-10-15 01:12:18 -07:00
Debanjum Singh Solanky
81fb65fa0a Return data sources to use if exception in data source chat actor 
Previously no value was returned if an exception got triggered when
collecting information sources to search.
 2024-10-14 18:20:20 -07:00
Debanjum Singh Solanky
3c93f07b3f Try respond even if web search, webpage read fails during chat 
Khoj shouldn't refuse to respond to user if web lookups fail.
It should transparently mention that online search etc. failed.
But try respond as best as it can without those references

This change ensures a response to the users query is attempted even
when web info retrieval fails.
 2024-10-14 18:13:26 -07:00
Debanjum Singh Solanky
07ab7ebf07 Try respond even if document search via inference endpoint fails 
The huggingface endpoint can be flaky. Khoj shouldn't refuse to
respond to user if document search fails.
It should transparently mention that document lookup failed.
But try respond as best as it can without the document references

This changes provides graceful failover when inference endpoint
requests fail either when encoding query or reranking retrieved docs
 2024-10-14 18:13:26 -07:00
Debanjum Singh Solanky
d6206aa80c Remove deprecated GET chat API endpoint 2024-10-14 18:13:26 -07:00
Debanjum Singh Solanky
263eee4351 Fix default chat model to use user model if no server chat model set 
- Advanced chat model should also fallback to user chat model if set
- Get conversation config should falback to user chat model if set

These assume no server chat model settings is configured
 2024-10-14 18:13:26 -07:00
Debanjum Singh Solanky
abcd11cfc0 Merge branch 'master' into features/advanced-reasoning 2024-10-13 03:06:23 -07:00
Debanjum Singh Solanky
9356e66b94 Fix default chat model to use user model if no server chat model set 
- Advanced chat model should also fallback to user chat model if set
- Get conversation config should falback to user chat model if set

These assume no server chat model settings is configured
 2024-10-13 03:02:29 -07:00
Debanjum Singh Solanky
9314f0a398 Fix default chat configs to use user model if no server chat model set 
Post merge cleanup in advanced reasoning to fallback to user chat
model if no server chat model defined for advanced and default
 2024-10-13 02:59:10 -07:00
Debanjum Singh Solanky
a2200466b7 Merge branch 'master' into features/advanced-reasoning 2024-10-12 21:01:22 -07:00
Debanjum Singh Solanky
9daaae0fdb Render inline any image files output by code in message 
Update regex to also include any links to code generated images that
aren't explicitly meant to be displayed inline. This allows folks to
download the image (unlike the fake link that doesn't work created by
model)
 2024-10-12 10:34:57 -07:00
Debanjum Singh Solanky
20d495c43a Update the iterative chat director prompt to generalize across chat models 
These prompts work across o1 and standard openai model. Works with
anthropic and google models as well
 2024-10-12 10:34:57 -07:00
sabaimran
01a58b71a5 Skip image, code generation if in research mode 2024-10-10 18:06:29 -07:00
Debanjum Singh Solanky
1b13d069f5 Pass data collected from various sources to code tool in normal flow too 2024-10-10 05:19:27 -07:00
Debanjum Singh Solanky
f462d34547 Render images files output by code interpreter in message on web app 2024-10-10 05:17:53 -07:00
Debanjum Singh Solanky
564491e164 Extract date filters quoted with non-ascii quotes in query 2024-10-10 04:45:00 -07:00
Debanjum Singh Solanky
6a8fd9bf33 Reorder embeddings search arguments based on argument importance 2024-10-10 04:45:00 -07:00
Debanjum Singh Solanky
0eacc0b2b0 Use consistent name for user, planner to not miss current user query 
Previously Khoj would start answering the previous query. This maybe
because the prompt uses User for prompt in chat history but was using
Q for current user prompt.
 2024-10-10 04:45:00 -07:00
Debanjum Singh Solanky
284c8c331b Increase default max iterations for research chat director to 5 2024-10-10 04:45:00 -07:00
Debanjum Singh Solanky
1e390325d2 Let research chat director decide which webpage to read, if any 
Make webpages to read automatically on search_online configurable via
a argument.

Set it to default to 1, so other callers of the function
are unaffected.

But iterative chat director can still decide which, if
any, webpages to read based on the online search it performs
 2024-10-10 04:45:00 -07:00
Debanjum Singh Solanky
5a699a52d2 Improve webpage summarization prompt to better extract links, excerpts 
This change allows the iterative director to dive deeper into its
research as the data extracted contains relevant links from the webpage

Previous summarization prompt didn't extract relevant links from the
webpage which limited further explorations from webpages
 2024-10-10 04:45:00 -07:00
Debanjum Singh Solanky
61df1d5db8 Pass previous iteration results to code interpreter chat actors 
This improves the code interpreter chat actors abilitiy to generate
code with data collected during the previous iterations
 2024-10-10 04:45:00 -07:00
Debanjum Singh Solanky
9e7025b330 Set python interpret sandbox url via environment variable 2024-10-10 04:45:00 -07:00
Debanjum Singh Solanky
2dc5804571 Extract defilter query into conversation utils for reuse 2024-10-10 04:45:00 -07:00
sabaimran
e69a8382f2 Add a code icon for code-related train of thought 2024-10-09 23:56:57 -07:00
sabaimran
536422a40c Include code snippets in the reference panel 2024-10-09 23:54:11 -07:00
Debanjum Singh Solanky
8d33c764b7 Allow iterative chat director to use python interpreter as a tool 2024-10-09 23:38:20 -07:00
Debanjum Singh Solanky
b373073f47 Show executed code in web app chat message references 2024-10-09 22:13:18 -07:00
Debanjum Singh Solanky
a98f97ed5e Refactor Run Code tool into separate module and modularize code functions 
Move construct_chat_history and ChatEvent enum into conversation.utils
and move send_message_to_model_wrapper to conversation.helper to
modularize code. And start thinning out the bloated routers.helper

- conversation.util components are shared functions that conversation
  child packages can use.
- conversation.helper components can't be imported by conversation
  packages but it can use these child packages

This division allows better modularity while avoiding circular
import dependencies
 2024-10-09 22:13:17 -07:00
Debanjum Singh Solanky
8044733201 Give Khoj ability to run python code as a tool triggered via chat API 
Create python code executing chat actor
- The chat actor generate python code within sandbox constraints
- Run the generated python code in the cohere terrarium, pyodide
  based sandbox accessible at sandbox url
 2024-10-09 21:37:22 -07:00
Debanjum Singh Solanky
4d33239af6 Improve prompts for the iterative chat director 2024-10-09 21:23:18 -07:00
Debanjum Singh Solanky
6ad85e2275 Fix to continue showing retrieved documents in train of thought 2024-10-09 21:20:22 -07:00
sabaimran
a6f6e4f418 Fix notes references and passage of user query in the chat flow 2024-10-09 20:34:20 -07:00
Debanjum Singh Solanky
ec248efd31 Allow iterative chat director to do notes search 2024-10-09 19:04:59 -07:00
Debanjum Singh Solanky
a6905a9f0c Pass background context to iterating chat director 2024-10-09 19:04:59 -07:00
sabaimran
028b6e6379 Fix yield for scraping direct web page 2024-10-09 18:14:08 -07:00
sabaimran
717d9da8d8 Handle when summarize result is not present, rename variable in for loop from query 2024-10-09 17:57:08 -07:00
sabaimran
03544efde2 Ignore typing of the result dict for online, web page scrape 2024-10-09 17:48:24 -07:00
sabaimran
ab81b01fcb Fix typing of direct_web_pages and remove the deprecated chat API 2024-10-09 17:46:28 -07:00
sabaimran
5b8d663cf1 Add intermediate summarization of results when planning with o1 2024-10-09 17:40:56 -07:00
sabaimran
7b288a1179 Clean up the function planning prompt a little bit 2024-10-09 16:59:20 -07:00
sabaimran
f71e4969d3 Skip summarize while it's broken, and snip some other parts of the workflow while under construction 2024-10-09 16:40:06 -07:00
sabaimran
f7e6f99a32 add typing for extract document references 2024-10-09 16:05:34 -07:00
sabaimran
6960fb097c update types of prev iterations response 2024-10-09 16:04:39 -07:00
sabaimran
4978360852 Fix type of previous_iterations 2024-10-09 16:02:41 -07:00
sabaimran
46ef205a75 Add additional type annotations for compiled_references et al 2024-10-09 16:01:52 -07:00
sabaimran
4fbaef10e9 Correct usage of the summarize function 2024-10-09 15:58:05 -07:00
sabaimran
c91678078d Correct the usage of query passed to summarize function 2024-10-09 15:55:55 -07:00
sabaimran
f867d5ed72 Working prototype of meta-level chain of reasoning and execution 
- Create a more dynamic reasoning agent that can evaluate information and understand what it doesn't know, making moves to get that information
- Lots of hacks and code that needs to be reversed later on before submission
 2024-10-09 15:54:25 -07:00
280 changed files with 21370 additions and 13859 deletions
Expand all files Collapse all files

13
.dockerignore
View File

@@ -1,10 +1,11 @@
.git/
.pytest_cache/
.vscode/
.venv/
docs/
.*
**/__pycache__/
*.egg-info/
documentation/
tests/
build/
dist/
scripts/
*.egg-info/
src/interface/
src/telemetry/
!src/interface/web

8
.github/workflows/desktop.yml vendored
View File

@@ -42,10 +42,10 @@ jobs:
run: |
npx todesktop build
- name: 📦 Release Desktop App
if: startsWith(github.ref, 'refs/tags/')
run: |
npx todesktop release --latest --force
# - name: 📦 Release Desktop App
# if: startsWith(github.ref, 'refs/tags/')
# run: |
# npx todesktop release --latest --force
- name: ⤵️ Get Desktop Apps
if: startsWith(github.ref, 'refs/tags/')

76
.github/workflows/dockerize.yml vendored
View File

@@ -38,13 +38,23 @@ env:
jobs:
build:
name: Publish Khoj Docker Images
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
image:
- 'local'
- 'cloud'
include:
- image: 'local'
platform: linux/amd64
runner: ubuntu-latest
- image: 'local'
platform: linux/arm64
runner: ubuntu-linux-arm64
- image: 'cloud'
platform: linux/amd64
runner: ubuntu-latest
- image: 'cloud'
platform: linux/arm64
runner: ubuntu-linux-arm64
runs-on: ${{ matrix.runner }}
steps:
- name: Checkout Code
uses: actions/checkout@v3
@@ -52,9 +62,6 @@ jobs:
# Get all history to correctly infer Khoj version using hatch
fetch-depth: 0
- name: 🧹 Delete huge unnecessary tools folder
run: rm -rf /opt/hostedtoolcache
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2
@@ -73,31 +80,70 @@ jobs:
run: rm -rf /opt/hostedtoolcache
- name: 📦 Build and Push Docker Image
uses: docker/build-push-action@v2
uses: docker/build-push-action@v4
if: (matrix.image == 'local' && github.event_name == 'workflow_dispatch') && github.event.inputs.khoj == 'true' || (matrix.image == 'local' && github.event_name == 'push')
with:
context: .
file: Dockerfile
platforms: linux/amd64, linux/arm64
push: true
tags: |
ghcr.io/${{ github.repository }}:${{ env.DOCKER_IMAGE_TAG }}
${{ github.ref_type == 'tag' && format('ghcr.io/{0}:latest', github.repository) || '' }}
ghcr.io/${{ github.repository }}:${{ env.DOCKER_IMAGE_TAG }}-${{ matrix.platform == 'linux/amd64' && 'amd64' || 'arm64' }}
build-args: |
VERSION=${{ steps.hatch.outputs.version }}
PORT=42110
cache-from: type=gha,scope=${{ matrix.image }}-${{ matrix.platform }}
cache-to: type=gha,mode=max,scope=${{ matrix.image }}-${{ matrix.platform}}
labels: |
org.opencontainers.image.description=Khoj AI - Your second brain powered by LLMs and Neural Search
org.opencontainers.image.source=${{ github.server_url }}/${{ github.repository }}
- name: 📦️⛅️ Build and Push Cloud Docker Image
uses: docker/build-push-action@v2
uses: docker/build-push-action@v4
if: (matrix.image == 'cloud' && github.event_name == 'workflow_dispatch') && github.event.inputs.khoj-cloud == 'true' || (matrix.image == 'cloud' && github.event_name == 'push')
with:
context: .
file: prod.Dockerfile
platforms: linux/amd64, linux/arm64
push: true
tags: |
ghcr.io/${{ github.repository }}-cloud:${{ env.DOCKER_IMAGE_TAG }}
${{ github.ref_type == 'tag' && format('ghcr.io/{0}-cloud:latest', github.repository) || '' }}
ghcr.io/${{ github.repository }}-cloud:${{ env.DOCKER_IMAGE_TAG }}-${{ matrix.platform == 'linux/amd64' && 'amd64' || 'arm64' }}
build-args: |
VERSION=${{ steps.hatch.outputs.version }}
PORT=42110
cache-from: type=gha,scope=${{ matrix.image }}-${{ matrix.platform }}
cache-to: type=gha,mode=max,scope=${{ matrix.image }}-${{ matrix.platform}}
labels: |
org.opencontainers.image.description=Khoj AI Cloud - Your second brain powered by LLMs and Neural Search
org.opencontainers.image.source=${{ github.server_url }}/${{ github.repository }}
manifest:
needs: build
runs-on: ubuntu-latest
if: github.event_name != 'pull_request'
steps:
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Login to GitHub Container Registry
uses: docker/login-action@v2
with:
registry: ghcr.io
username: ${{ github.repository_owner }}
password: ${{ secrets.PAT }}
- name: Create and Push Local Manifest
if: github.event.inputs.khoj == 'true' || github.event_name == 'push'
run: |
docker buildx imagetools create \
-t ghcr.io/${{ github.repository }}:${{ env.DOCKER_IMAGE_TAG }} \
-t ghcr.io/${{ github.repository }}:${{ github.ref_type == 'tag' && 'latest' || env.DOCKER_IMAGE_TAG }} \
ghcr.io/${{ github.repository }}:${{ env.DOCKER_IMAGE_TAG }}-amd64 \
ghcr.io/${{ github.repository }}:${{ env.DOCKER_IMAGE_TAG }}-arm64
- name: Create and Push Cloud Manifest
if: github.event.inputs.khoj-cloud == 'true' || github.event_name == 'push'
run: |
docker buildx imagetools create \
-t ghcr.io/${{ github.repository }}-cloud:${{ env.DOCKER_IMAGE_TAG }} \
-t ghcr.io/${{ github.repository }}-cloud:${{ github.ref_type == 'tag' && 'latest' || env.DOCKER_IMAGE_TAG }} \
ghcr.io/${{ github.repository }}-cloud:${{ env.DOCKER_IMAGE_TAG }}-amd64 \
ghcr.io/${{ github.repository }}-cloud:${{ env.DOCKER_IMAGE_TAG }}-arm64

5
.github/workflows/pypi.yml vendored
View File

@@ -49,7 +49,7 @@ jobs:
- name: 📂 Copy Generated Files
run: |
mkdir -p src/khoj/interface/compiled
cp -r /opt/hostedtoolcache/Python/3.11.10/x64/lib/python3.11/site-packages/khoj/interface/compiled/* src/khoj/interface/compiled/
cp -r /opt/hostedtoolcache/Python/3.11.11/x64/lib/python3.11/site-packages/khoj/interface/compiled/* src/khoj/interface/compiled/
- name: ⚙️ Build Python Package
run: |
@@ -75,6 +75,7 @@ jobs:
- name: 📦 Publish Python Package to PyPI
if: startsWith(github.ref, 'refs/tags') || github.ref == 'refs/heads/master'
uses: pypa/gh-action-pypi-publish@v1.8.14
uses: pypa/gh-action-pypi-publish@release/v1.12
with:
skip-existing: true
print-hash: true

158
.github/workflows/run_evals.yml vendored Normal file
View File

@@ -0,0 +1,158 @@
name: eval
on:
# Run on every release
push:
tags:
- "*"
# Allow manual triggers from GitHub UI
workflow_dispatch:
inputs:
khoj_mode:
description: 'Khoj Mode (general/default/research)'
required: true
default: 'default'
type: choice
options:
- general
- default
- research
dataset:
description: 'Dataset to evaluate (frames/simpleqa)'
required: true
default: 'frames'
type: choice
options:
- frames
- simpleqa
- gpqa
- math500
sample_size:
description: 'Number of samples to evaluate'
required: false
default: 200
type: number
jobs:
eval:
runs-on: ubuntu-latest
strategy:
matrix:
# Use input from manual trigger if available, else run all combinations
khoj_mode: ${{ github.event_name == 'workflow_dispatch' && fromJSON(format('["{0}"]', inputs.khoj_mode)) || fromJSON('["general", "default", "research"]') }}
dataset: ${{ github.event_name == 'workflow_dispatch' && fromJSON(format('["{0}"]', inputs.dataset)) || fromJSON('["frames", "simpleqa"]') }}
services:
postgres:
image: ankane/pgvector
env:
POSTGRES_PASSWORD: postgres
POSTGRES_USER: postgres
POSTGRES_DB: postgres
ports:
- 5432:5432
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
steps:
- uses: actions/checkout@v3
with:
fetch-depth: 0
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: '3.10'
- name: Get App Version
id: hatch
run: echo "version=$(pipx run hatch version)" >> $GITHUB_OUTPUT
- name: ⏬️ Install Dependencies
env:
DEBIAN_FRONTEND: noninteractive
run: |
# install postgres and other dependencies
sudo apt update && sudo apt install -y git python3-pip libegl1 sqlite3 libsqlite3-dev libsqlite3-0 ffmpeg libsm6 libxext6
sudo apt install -y postgresql postgresql-client && sudo apt install -y postgresql-server-dev-14
# upgrade pip
python -m ensurepip --upgrade && python -m pip install --upgrade pip
# install terrarium for code sandbox
git clone https://github.com/khoj-ai/terrarium.git && cd terrarium && npm install --legacy-peer-deps && mkdir pyodide_cache
- name: ⬇️ Install Application
run: |
sed -i 's/dynamic = \["version"\]/version = "${{ steps.hatch.outputs.version }}"/' pyproject.toml
pip install --upgrade .[dev]
- name: 📝 Run Eval
env:
KHOJ_MODE: ${{ matrix.khoj_mode }}
SAMPLE_SIZE: ${{ github.event_name == 'workflow_dispatch' && inputs.sample_size || 200 }}
BATCH_SIZE: "20"
RANDOMIZE: "True"
KHOJ_URL: "http://localhost:42110"
KHOJ_LLM_SEED: "42"
GEMINI_API_KEY: ${{ secrets.GEMINI_API_KEY }}
SERPER_DEV_API_KEY: ${{ matrix.dataset != 'math500' && secrets.SERPER_DEV_API_KEY }}
OLOSTEP_API_KEY: ${{ matrix.dataset != 'math500' && secrets.OLOSTEP_API_KEY }}
HF_TOKEN: ${{ secrets.HF_TOKEN }}
KHOJ_ADMIN_EMAIL: khoj
KHOJ_ADMIN_PASSWORD: khoj
POSTGRES_HOST: localhost
POSTGRES_PORT: 5432
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: postgres
KHOJ_TELEMETRY_DISABLE: "True" # To disable telemetry for tests
run: |
# Start Khoj server in background
khoj --anonymous-mode --non-interactive &
# Start code sandbox
npm install -g pm2
npm run ci --prefix terrarium
# Wait for server to be ready
timeout=120
while ! curl -s http://localhost:42110/api/health > /dev/null; do
if [ $timeout -le 0 ]; then
echo "Timed out waiting for Khoj server"
exit 1
fi
echo "Waiting for Khoj server..."
sleep 2
timeout=$((timeout-2))
done
# Run evals
python tests/evals/eval.py -d ${{ matrix.dataset }}
- name: Upload Results
if: always() # Upload results even if tests fail
uses: actions/upload-artifact@v3
with:
name: eval-results-${{ steps.hatch.outputs.version }}-${{ matrix.khoj_mode }}-${{ matrix.dataset }}
path: |
*_evaluation_results_*.csv
*_evaluation_summary_*.txt
- name: Display Results
if: always()
run: |
# Read and display summary
echo "## Evaluation Summary of Khoj on ${{ matrix.dataset }} in ${{ matrix.khoj_mode }} mode" >> $GITHUB_STEP_SUMMARY
echo "**$(head -n 1 *_evaluation_summary_*.txt)**" >> $GITHUB_STEP_SUMMARY
echo "- Khoj Version: ${{ steps.hatch.outputs.version }}" >> $GITHUB_STEP_SUMMARY
echo "- Chat Model: Gemini 1.5 Flash 002" >> $GITHUB_STEP_SUMMARY
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
tail -n +2 *_evaluation_summary_*.txt >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
# Display in logs too
echo "===== EVALUATION RESULTS ====="
cat *_evaluation_summary_*.txt

13
.github/workflows/test.yml vendored
View File

@@ -5,6 +5,7 @@ on:
paths:
- src/khoj/**
- tests/**
- '!tests/evals/**'
- config/**
- pyproject.toml
- .pre-commit-config.yml
@@ -15,6 +16,7 @@ on:
paths:
- src/khoj/**
- tests/**
- '!tests/evals/**'
- config/**
- pyproject.toml
- .pre-commit-config.yml
@@ -53,15 +55,13 @@ jobs:
with:
python-version: ${{ matrix.python_version }}
- name: Install Git
run: |
apt update && apt install -y git
- name: ⏬️ Install Dependencies
env:
DEBIAN_FRONTEND: noninteractive
run: |
apt update && apt install -y libegl1 sqlite3 libsqlite3-dev libsqlite3-0 ffmpeg libsm6 libxext6
apt update && apt install -y git libegl1 sqlite3 libsqlite3-dev libsqlite3-0 ffmpeg libsm6 libxext6
# required by llama-cpp-python prebuilt wheels
apt install -y musl-dev && ln -s /usr/lib/x86_64-linux-musl/libc.so /lib/libc.musl-x86_64.so.1
- name: ⬇️ Install Postgres
env:
@@ -76,6 +76,9 @@ jobs:
python -m pip install --upgrade pip
- name: ⬇️ Install Application
env:
PIP_EXTRA_INDEX_URL: "https://download.pytorch.org/whl/cpu https://abetlen.github.io/llama-cpp-python/whl/cpu"
CUDA_VISIBLE_DEVICES: ""
run: sed -i 's/dynamic = \["version"\]/version = "0.0.0"/' pyproject.toml && pip install --upgrade .[dev]
- name: 🧪 Test Application

10
.gitignore vendored
View File

@@ -42,3 +42,13 @@ src/interface/obsidian/main.js
# obsidian
data.json
# Android
src/interface/android/.gradle
src/interface/android/app/build
src/interface/android/build
src/interface/android/*.aab
src/interface/android/*.apk
src/interface/android/*.apk.idsig
src/interface/android/*.keystore
src/interface/android/local.properties

2
.pre-commit-config.yaml
View File

@@ -26,7 +26,7 @@ repos:
rev: v1.0.0
hooks:
- id: mypy
stages: [push, manual]
stages: [pre-push, manual]
pass_filenames: false
args:
- --config-file=pyproject.toml

49
Dockerfile
View File

@@ -1,5 +1,5 @@
# syntax=docker/dockerfile:1
FROM ubuntu:jammy
FROM ubuntu:jammy AS base
LABEL homepage="https://khoj.dev"
LABEL repository="https://github.com/khoj-ai/khoj"
LABEL org.opencontainers.image.source="https://github.com/khoj-ai/khoj"
@@ -13,36 +13,51 @@ RUN apt update -y && apt -y install \
# Required by RapidOCR
libgl1 \
libglx-mesa0 \
libglib2.0-0 && \
# Required by Next.js Web app
curl -sL https://deb.nodesource.com/setup_20.x | bash - && \
curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add - && \
echo "deb https://dl.yarnpkg.com/debian/ stable main" | tee /etc/apt/sources.list.d/yarn.list && \
apt update -y && apt -y --no-install-recommends install nodejs yarn && \
libglib2.0-0 \
# Required by llama-cpp-python pre-built wheels. See #1628
musl-dev && \
ln -s /usr/lib/x86_64-linux-musl/libc.so /lib/libc.musl-x86_64.so.1 && \
# Clean up
apt clean && rm -rf /var/lib/apt/lists/*
# Install Application
# Build Server
FROM base AS server-deps
WORKDIR /app
COPY pyproject.toml .
COPY README.md .
ARG VERSION=0.0.0
# use the pre-built llama-cpp-python, torch cpu wheel
ENV PIP_EXTRA_INDEX_URL="https://download.pytorch.org/whl/cpu https://abetlen.github.io/llama-cpp-python/whl/cpu"
# avoid downloading unused cuda specific python packages
ENV CUDA_VISIBLE_DEVICES=""
RUN sed -i "s/dynamic = \\[\"version\"\\]/version = \"$VERSION\"/" pyproject.toml && \
pip install --no-cache-dir .
# Copy Source Code
COPY . .
# Set the PYTHONPATH environment variable in order for it to find the Django app.
ENV PYTHONPATH=/app/src:$PYTHONPATH
# Go to the directory src/interface/web and export the built Next.js assets
# Build Web App
FROM node:20-alpine AS web-app
# Set build optimization env vars
ENV NODE_ENV=production
ENV NEXT_TELEMETRY_DISABLED=1
WORKDIR /app/src/interface/web
RUN bash -c "yarn install --frozen-lockfile --verbose && yarn ciexport && yarn cache clean"
# Install dependencies first (cache layer)
COPY src/interface/web/package.json src/interface/web/yarn.lock ./
RUN yarn install --frozen-lockfile
# Copy source and build
COPY src/interface/web/. ./
RUN yarn build
# Merge the Server and Web App into a Single Image
FROM base
ENV PYTHONPATH=/app/src
WORKDIR /app
COPY --from=server-deps /usr/local/lib/python3.10/dist-packages /usr/local/lib/python3.10/dist-packages
COPY --from=web-app /app/src/interface/web/out ./src/khoj/interface/built
COPY . .
RUN cd src && python3 khoj/manage.py collectstatic --noinput
# Run the Application
# There are more arguments required for the application to run,
# but these should be passed in through the docker-compose.yml file.
# but those should be passed in through the docker-compose.yml file.
ARG PORT
EXPOSE ${PORT}
ENTRYPOINT ["python3", "src/khoj/main.py"]

13
README.md
View File

@@ -1,4 +1,4 @@
<p align="center"><img src="src/khoj/interface/web/assets/icons/khoj-logo-sideways-500.png" width="230" alt="Khoj Logo"></p>
<p align="center"><img src="https://assets.khoj.dev/khoj-logo-sideways-1200x540.png" width="230" alt="Khoj Logo"></p>
<div align="center">
@@ -29,10 +29,17 @@
</div>
<div align="left">
***
### 🎁 New
* Start any message with `/research` to try out the experimental research mode with Khoj.
* Anyone can now [create custom agents](https://blog.khoj.dev/posts/create-agents-on-khoj/) with tunable personality, tools and knowledge bases.
* [Read](https://blog.khoj.dev/posts/evaluate-khoj-quality/) about Khoj's excellent performance on modern retrieval and reasoning benchmarks.
***
## Overview
[Khoj](https://khoj.dev) is a personal AI app to extend your capabilities. It smoothly scales up from an on-device personal AI to a cloud-scale enterprise AI.
- Chat with any local or online LLM (e.g llama3, qwen, gemma, mistral, gpt, claude, gemini).
@@ -47,8 +54,6 @@
***
</div>
## See it in action
![demo_chat](https://github.com/khoj-ai/khoj/blob/master/documentation/assets/img/quadratic_equation_khoj_web.gif?raw=true)

46
docker-compose.yml
View File

@@ -14,13 +14,25 @@ services:
interval: 30s
timeout: 10s
retries: 5
sandbox:
image: ghcr.io/khoj-ai/terrarium:latest
ports:
- "8080:8080"
search:
image: docker.io/searxng/searxng:latest
ports:
- "42113:8080"
volumes:
- khoj_search:/etc/searxng
environment:
- SEARXNG_BASE_URL=http://localhost:42113/
server:
depends_on:
database:
condition: service_healthy
# Use the following line to use the latest version of khoj. Otherwise, it will build from source.
# Use the following line to use the latest version of khoj. Otherwise, it will build from source. Set this to ghcr.io/khoj-ai/khoj-cloud:latest if you want to use the prod image.
image: ghcr.io/khoj-ai/khoj:latest
# Uncomment the following line to build from source. This will take a few minutes. Comment the next two lines out if you want to use the offiicial image.
# Uncomment the following line to build from source. This will take a few minutes. Comment the next two lines out if you want to use the official image.
# build:
# context: .
ports:
@@ -33,6 +45,7 @@ services:
volumes:
- khoj_config:/root/.khoj/
- khoj_models:/root/.cache/torch/sentence_transformers
- khoj_models:/root/.cache/huggingface
# Use 0.0.0.0 to explicitly set the host ip for the service on the container. https://pythonspeed.com/articles/docker-connection-refused/
environment:
- POSTGRES_DB=postgres
@@ -44,12 +57,33 @@ services:
- KHOJ_DEBUG=False
- KHOJ_ADMIN_EMAIL=username@example.com
- KHOJ_ADMIN_PASSWORD=password
# Uncomment lines below to use chat models by each provider.
# Default URL of Terrarium, the Python sandbox used by Khoj to run code. Its container is specified above
- KHOJ_TERRARIUM_URL=http://host.docker.internal:8080
# Default URL of SearxNG, the default web search engine used by Khoj. Its container is specified above
- KHOJ_SEARXNG_URL=http://host.docker.internal:42113
# Uncomment line below to use with Ollama running on your local machine at localhost:11434.
# Change URL to use with other OpenAI API compatible providers like VLLM, LMStudio etc.
# - OPENAI_API_BASE=http://host.docker.internal:11434/v1/
#
# Uncomment appropriate lines below to use chat models by OpenAI, Anthropic, Google.
# Ensure you set your provider specific API keys.
# ---
# - OPENAI_API_KEY=your_openai_api_key
# - GEMINI_API_KEY=your_gemini_api_key
# - ANTHROPIC_API_KEY=your_anthropic_api_key
#
# Uncomment appropriate lines below to enable web results with Khoj
# Ensure you set your provider specific API keys.
# ---
# Free, Slower API. Does both web search and webpage read. Get API key from https://jina.ai/
# - JINA_API_KEY=your_jina_api_key
# Paid, Fast API. Only does web search. Get API key from https://serper.dev/
# - SERPER_DEV_API_KEY=your_serper_dev_api_key
# Paid, Fast, Open API. Only does webpage read. Get API key from https://firecrawl.dev/
# - FIRECRAWL_API_KEY=your_firecrawl_api_key
# Paid, Fast, Higher Read Success API. Only does webpage read. Get API key from https://olostep.com/
# - OLOSTEP_API_KEY=your_olostep_api_key
#
# Uncomment the necessary lines below to make your instance publicly accessible.
# Replace the KHOJ_DOMAIN with either your domain or IP address (no http/https prefix).
# Proceed with caution, especially if you are using anonymous mode.
@@ -57,6 +91,11 @@ services:
# - KHOJ_NO_HTTPS=True
# - KHOJ_DOMAIN=192.168.0.104
# - KHOJ_DOMAIN=khoj.example.com
# Uncomment the line below to disable telemetry.
# Telemetry helps us prioritize feature development and understand how people are using Khoj
# Read more at https://docs.khoj.dev/miscellaneous/telemetry
# - KHOJ_TELEMETRY_DISABLE=True
# Comment out this line when you're using the official ghcr.io/khoj-ai/khoj-cloud:latest prod image.
command: --host="0.0.0.0" --port=42110 -vv --anonymous-mode --non-interactive
@@ -64,3 +103,4 @@ volumes:
khoj_config:
khoj_db:
khoj_models:
khoj_search:

BIN
documentation/assets/img/chrome_pwa_alt.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 500 KiB

BIN
documentation/assets/img/khoj_chat_on_desktop.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 298 KiB

BIN
documentation/assets/img/khoj_documentation.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 868 KiB

BIN
documentation/assets/img/khoj_search_on_desktop.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 333 KiB

BIN
documentation/assets/img/magic_link.png
View File

Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 71 KiB

After

Width:  |  Height:  |  Size: 23 KiB

BIN
documentation/assets/img/mic_chat_icon.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 8.4 KiB

BIN
documentation/assets/img/pwa_install_desktop.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 15 KiB

11
documentation/docs/advanced/admin.md
View File

@@ -1,6 +1,8 @@
# Admin Panel
> Describes the Khoj settings configurable via the admin panel
By default, you admin panel is available at `http://localhost:42110/server/admin/`. You can access the admin panel by logging in with your admin credentials (this would be your `KHOJ_ADMIN_EMAIL` and `KHOJ_ADMIN_PASSWORD`). The admin panel allows you to configure various settings for your Khoj server.
## App Settings
### Agents
Add all the agents you want to use for your different use-cases like Writer, Researcher, Therapist etc.
@@ -36,14 +38,13 @@ To add a server chat setting:
- The `Advanced` field doesn't need to be set when self-hosting. When unset, the `Default` chat model is used for all users and the intermediate steps.
### OpenAI Processor Conversation Configs
These settings configure chat model providers to be accessed over API.
The name of this setting is kind of a misnomer, we know, it'll hopefully be changed at some point.
For each chat model provider you [add](http://localhost:42110/server/admin/database/openaiprocessorconversationconfig/add):
### AI Model API
These settings configure APIs to interact with AI models.
For each AI Model API you [add](http://localhost:42110/server/admin/database/aimodelapi/add):
- `Api key`: Set to your [OpenAI](https://platform.openai.com/api-keys), [Anthropic](https://console.anthropic.com/account/keys) or [Gemini](https://aistudio.google.com/app/apikey) API keys.
- `Name`: Give the configuration any friendly name like `OpenAI`, `Gemini`, `Anthropic`.
- `Api base url`: Set the API base URL. This is only relevant to set if you're using another OpenAI-compatible proxy server like [Ollama](/advanced/ollama) or [LMStudio](/advanced/lmstudio).
![example configuration for openai processor](/img/example_openai_processor_config.png)
![example configuration for ai model api](/img/example_openai_processor_config.png)
### Search Model Configs
Search models are used to generate vector embeddings of your documents for natural language search and chat. You can choose any [embeddings models on HuggingFace](https://huggingface.co/models?pipeline_tag=sentence-similarity) to try, use for your to create vector embeddings of your documents for natural language search and chat.

52
documentation/docs/advanced/authentication.md
View File

@@ -1,52 +0,0 @@
# Authenticate
:::info
This is only helpful for self-hosted users or teams. If you're using [Khoj Cloud](https://app.khoj.dev), both Magic Links and Google OAuth work.
:::
By default, most of the instructions for self-hosting Khoj assume a single user, and so the default configuration is to run in anonymous mode. However, if you want to enable authentication, you can do so either with with [Magic Links](#using-magic-links) or [Google OAuth](#using-google-oauth) as shown below. This can be helpful to make Khoj securely accessible to you and your team.
:::tip[Note]
Remove the `--anonymous-mode` flag from your khoj start up command or docker-compose file to enable authentication.
:::
## Using Magic Links
The most secure way to do this is to integrate with [Resend](https://resend.com) by setting up an account and adding an environment variable for `RESEND_API_KEY`. You can get your API key [here](https://resend.com/api-keys). This will allow you to automatically send sign-in links to users who want to log in.
It's still possible to use the magic links feature without Resend, but you'll need to manually send the magic links to users who want to log in.
## Manually sending magic links
1. The user will have to enter their email address in the login form.
They'll click `Send Magic Link`. Without the Resend API key, this will just create an unverified account for them in the backend
<img src="/img/magic_link.png" alt="Magic link login form" width="400"/>
2. You can get their magic link using the admin panel
Go to the [admin panel](http://localhost:42110/server/admin/database/khojuser/). You'll see a list of users. Search for the user you want to send a magic link to. Tick the checkbox next to their row, and use the action drop down at the top to 'Get email login URL'. This will generate a magic link that you can send to the user, which will appear at the top of the admin interface.
| Get email login URL | Retrieved login URL |
|---------------------|---------------------|
| <img src="/img/admin_get_emali_login.png" alt="Get user magic sign in link" width="400" />| <img src="/img/admin_successful_login_url.png" alt="Successfully retrieved a login URL" width="400" />|
3. Send the magic link to the user. They can click on it to log in.
Once they click on the link, they'll automatically be logged in. They'll have to repeat this process for every new device they want to log in from, but they shouldn't have to repeat it on the same device.
A given magic link can only be used once. If the user tries to use it again, they'll be redirected to the login page to get a new magic link.
## Using Google OAuth
To set up your self-hosted Khoj with Google Auth, you need to create a project in the Google Cloud Console and enable the Google Auth API.
To implement this, you'll need to:
1. You must use the `python` package or build from source, because you'll need to install additional packages for the google auth libraries (`prod`). The syntax to install the right packages is
```
pip install khoj[prod]
```
2. [Create authorization credentials](https://developers.google.com/identity/sign-in/web/sign-in) for your application.
3. Open your [Google cloud console](https://console.developers.google.com/apis/credentials) and create a configuration like below for the relevant `OAuth 2.0 Client IDs` project:
![Google auth login project settings](https://github.com/khoj-ai/khoj/assets/65192171/9bcbf6f4-197d-4d0c-973a-c10b1331c892)
4. Configure these environment variables: `GOOGLE_CLIENT_SECRET`, and `GOOGLE_CLIENT_ID`. You can find these values in the Google cloud console, in the same place where you configured the authorized origins and redirect URIs.
That's it! That should be all you have to do. Now, when you reload Khoj without `--anonymous-mode`, you should be able to use your Google account to sign in.

74
documentation/docs/advanced/authentication.mdx Normal file
View File

@@ -0,0 +1,74 @@
# Authenticate (Multi-User Setup)
```mdx-code-block
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
```
By default, most of the instructions for self-hosting Khoj assume a single user, and so the default configuration is to run in anonymous mode. However, if you want to enable authentication, you can do so either with with [Magic Links](#using-magic-links) or [Google OAuth](#using-google-oauth) as shown below. This can be helpful to make Khoj securely accessible to you and your team.
:::tip[Note]
Remove the `--anonymous-mode` flag from your khoj start up command or docker-compose file to enable authentication.
:::
## Using Magic Links
The most secure way to do this is to integrate with [Resend](https://resend.com).
1. Setup your account at https://resend.com
2. Set an environment variable for `RESEND_API_KEY`. You can get your API key [here](https://resend.com/api-keys).
3. Set an environment variable for `RESEND_EMAIL`. This is the email address that will show up in your `from` field when sending magic links.
This will allow you to automatically send sign-in links to users who want to log in.
It's still possible to use the magic links feature without Resend, but you'll need to manually send the magic links to users who want to log in.
## Manually sending magic links
1. The user will have to enter their email address in the login page at http://localhost:42110/login.
They'll click `Get Login Link`. Without the Resend API key, this will just create an unverified account for them in the backend
<img src="/img/magic_link.png" alt="Magic link login form" width="400"/>
2. You can get their magic link using the admin panel
Go to the [admin panel](http://localhost:42110/server/admin/database/khojuser/). You'll see a list of users. Search for the user you want to send a magic link to. Tick the checkbox next to their row, and use the action drop down at the top to 'Get email login URL'. This will generate a magic link that you can send to the user, which will appear at the top of the admin interface.
| Get email login URL | Retrieved login URL |
|---------------------|---------------------|
| <img src="/img/admin_get_emali_login.png" alt="Get user magic sign in link" width="400" />| <img src="/img/admin_successful_login_url.png" alt="Successfully retrieved a login URL" width="400" />|
3. Send the magic link to the user. They can click on it to log in.
Once they click on the link, they'll automatically be logged in. They'll have to repeat this process for every new device they want to log in from, but they shouldn't have to repeat it on the same device.
A given magic link can only be used once. If the user tries to use it again, they'll be redirected to the login page to get a new magic link.
## Using Google OAuth
For this method, you'll need to use the prod version of the Khoj package. You can install it as below:
<Tabs groupId="server" queryString>
<TabItem value="docker" label="Docker">
Update your `docker-compose.yml` to use the prod image
```bash
image: ghcr.io/khoj-ai/khoj-cloud:latest
```
</TabItem>
<TabItem value="pip" label="Pip">
```bash
pip install khoj[prod]
```
</TabItem>
</Tabs>
To set up your self-hosted Khoj with Google Auth, you need to create a project in the Google Cloud Console and enable the Google Auth API.
To implement this, you'll need to:
1. [Create authorization credentials](https://developers.google.com/identity/sign-in/web/sign-in) for your application.
2. Open your [Google cloud console](https://console.developers.google.com/apis/credentials) and create a configuration like below for the relevant `OAuth 2.0 Client IDs` project:
![Google auth login project settings](https://github.com/khoj-ai/khoj/assets/65192171/9bcbf6f4-197d-4d0c-973a-c10b1331c892)
3. Configure these environment variables: `GOOGLE_CLIENT_SECRET`, and `GOOGLE_CLIENT_ID`. You can find these values in the Google cloud console, in the same place where you configured the authorized origins and redirect URIs.
That's it! That should be all you have to do. Now, when you reload Khoj without `--anonymous-mode`, you should be able to use your Google account to sign in.

9
documentation/docs/advanced/litellm.md
View File

@@ -21,17 +21,14 @@ Using LiteLLM with Khoj makes it possible to turn any LLM behind an API into you
export MISTRAL_API_KEY=<MISTRAL_API_KEY>
litellm --model mistral/mistral-tiny --drop_params
```
3. Create a new [OpenAI Processor Conversation Config](http://localhost:42110/server/admin/database/openaiprocessorconversationconfig/add) on your Khoj admin panel
3. Create a new [API Model API](http://localhost:42110/server/admin/database/aimodelapi/add) on your Khoj admin panel
- Name: `proxy-name`
- Api Key: `any string`
- Api Base Url: **URL of your Openai Proxy API**
4. Create a new [Chat Model Option](http://localhost:42110/server/admin/database/chatmodeloptions/add) on your Khoj admin panel.
4. Create a new [Chat Model](http://localhost:42110/server/admin/database/chatmodel/add) on your Khoj admin panel.
- Name: `llama3.1` (replace with the name of your local model)
- Model Type: `Openai`
- Openai Config: `<the proxy config you created in step 3>`
- Max prompt size: `20000` (replace with the max prompt size of your model)
- Tokenizer: *Do not set for OpenAI, Mistral, Llama3 based models*
5. Create a new [Server Chat Setting](http://localhost:42110/server/admin/database/serverchatsettings/add/) on your Khoj admin panel
- Default model: `<name of chat model option you created in step 4>`
- Summarizer model: `<name of chat model option you created in step 4>`
6. Go to [your config](http://localhost:42110/settings) and select the model you just created in the chat model dropdown.
5. Go to [your config](http://localhost:42110/settings) and select the model you just created in the chat model dropdown.

7
documentation/docs/advanced/lmstudio.md
View File

@@ -18,13 +18,10 @@ LM Studio can expose an [OpenAI API compatible server](https://lmstudio.ai/docs/
- Name: `proxy-name`
- Api Key: `any string`
- Api Base Url: `http://localhost:1234/v1/` (default for LMStudio)
4. Create a new [Chat Model Option](http://localhost:42110/server/admin/database/chatmodeloptions/add) on your Khoj admin panel.
4. Create a new [Chat Model](http://localhost:42110/server/admin/database/chatmodel/add) on your Khoj admin panel.
- Name: `llama3.1` (replace with the name of your local model)
- Model Type: `Openai`
- Openai Config: `<the proxy config you created in step 3>`
- Max prompt size: `20000` (replace with the max prompt size of your model)
- Tokenizer: *Do not set for OpenAI, mistral, llama3 based models*
5. Create a new [Server Chat Setting](http://localhost:42110/server/admin/database/serverchatsettings/add/) on your Khoj admin panel
- Default model: `<name of chat model option you created in step 4>`
- Summarizer model: `<name of chat model option you created in step 4>`
6. Go to [your config](http://localhost:42110/settings) and select the model you just created in the chat model dropdown.
5. Go to [your config](http://localhost:42110/settings) and select the model you just created in the chat model dropdown.

36
documentation/docs/advanced/ollama.md
View File

@@ -1,36 +0,0 @@
# Ollama
:::info
This is only helpful for self-hosted users. If you're using [Khoj Cloud](https://app.khoj.dev), you're limited to our first-party models.
:::
:::info
Khoj natively supports local LLMs [available on HuggingFace in GGUF format](https://huggingface.co/models?library=gguf). Using an OpenAI API proxy with Khoj maybe useful for ease of setup, trying new models or using commercial LLMs via API.
:::
Ollama allows you to run [many popular open-source LLMs](https://ollama.com/library) locally from your terminal.
For folks comfortable with the terminal, Ollama's terminal based flows can ease setup and management of chat models.
Ollama exposes a local [OpenAI API compatible server](https://github.com/ollama/ollama/blob/main/docs/openai.md#models). This makes it possible to use chat models from Ollama to create your personal AI agents with Khoj.
## Setup
1. Setup Ollama: https://ollama.com/
2. Start your preferred model with Ollama. For example,
```bash
ollama run llama3.1
```
3. Create a new [OpenAI Processor Conversation Config](http://localhost:42110/server/admin/database/openaiprocessorconversationconfig/add) on your Khoj admin panel
- Name: `ollama`
- Api Key: `any string`
- Api Base Url: `http://localhost:11434/v1/` (default for Ollama)
4. Create a new [Chat Model Option](http://localhost:42110/server/admin/database/chatmodeloptions/add) on your Khoj admin panel.
- Name: `llama3.1` (replace with the name of your local model)
- Model Type: `Openai`
- Openai Config: `<the ollama config you created in step 3>`
- Max prompt size: `20000` (replace with the max prompt size of your model)
5. Create a new [Server Chat Setting](http://localhost:42110/server/admin/database/serverchatsettings/add/) on your Khoj admin panel
- Default model: `<name of chat model option you created in step 4>`
- Summarizer model: `<name of chat model option you created in step 4>`
6. Go to [your config](http://localhost:42110/settings) and select the model you just created in the chat model dropdown.
That's it! You should now be able to chat with your Ollama model from Khoj. If you want to add additional models running on Ollama, repeat step 6 for each model.

78
documentation/docs/advanced/ollama.mdx Normal file
View File

@@ -0,0 +1,78 @@
# Ollama
```mdx-code-block
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
```
:::info
This is only helpful for self-hosted users. If you're using [Khoj Cloud](https://app.khoj.dev), you can use our first-party supported models.
:::
:::info
Khoj can directly run local LLMs [available on HuggingFace in GGUF format](https://huggingface.co/models?library=gguf). The integration with Ollama is useful to run Khoj on Docker and have the chat models use your GPU or to try new models via CLI.
:::
Ollama allows you to run [many popular open-source LLMs](https://ollama.com/library) locally from your terminal.
For folks comfortable with the terminal, Ollama's terminal based flows can ease setup and management of chat models.
Ollama exposes a local [OpenAI API compatible server](https://github.com/ollama/ollama/blob/main/docs/openai.md#models). This makes it possible to use chat models from Ollama with Khoj.
## Setup
:::info
Restart your Khoj server after first run or update to the settings below to ensure all settings are applied correctly.
:::
<Tabs groupId="type" queryString>
<TabItem value="first-run" label="First Run">
<Tabs groupId="server" queryString>
<TabItem value="docker" label="Docker">
1. Setup Ollama: https://ollama.com/
2. Download your preferred chat model with Ollama. For example,
```bash
ollama pull llama3.1
```
3. Uncomment `OPENAI_API_BASE` environment variable in your downloaded Khoj [docker-compose.yml](https://github.com/khoj-ai/khoj/blob/master/docker-compose.yml#:~:text=OPENAI_API_BASE)
4. Start Khoj docker for the first time to automatically integrate and load models from the Ollama running on your host machine
```bash
# run below command in the directory where you downloaded the Khoj docker-compose.yml
docker-compose up
```
</TabItem>
<TabItem value="pip" label="Pip">
1. Setup Ollama: https://ollama.com/
2. Download your preferred chat model with Ollama. For example,
```bash
ollama pull llama3.1
```
3. Set `OPENAI_API_BASE` environment variable to `http://localhost:11434/v1/` in your shell before starting Khoj for the first time
```bash
export OPENAI_API_BASE="http://localhost:11434/v1/"
khoj --anonymous-mode
```
</TabItem>
</Tabs>
</TabItem>
<TabItem value="update" label="Update">
1. Setup Ollama: https://ollama.com/
2. Download your preferred chat model with Ollama. For example,
```bash
ollama pull llama3.1
```
3. Create a new [AI Model API](http://localhost:42110/server/admin/database/aimodelapi/add) on your Khoj admin panel
- Name: `ollama`
- Api Key: `any string`
- Api Base Url: `http://localhost:11434/v1/` (default for Ollama)
4. Create a new [Chat Model](http://localhost:42110/server/admin/database/chatmodel/add) on your Khoj admin panel.
- Name: `llama3.1` (replace with the name of your local model)
- Model Type: `Openai`
- Openai Config: `<the ollama config you created in step 3>`
- Max prompt size: `20000` (replace with the max prompt size of your model)
5. Go to [your config](http://localhost:42110/settings) and select the model you just created in the chat model dropdown.
If you want to add additional models running on Ollama, repeat step 4 for each model.
</TabItem>
</Tabs>
That's it! You should now be able to chat with your Ollama model from Khoj.

39
documentation/docs/advanced/tailscale.md Normal file
View File

@@ -0,0 +1,39 @@
# Tailscale
:::info
This is only helpful for secure cross-device access to **self-hosted** Khoj. You **do not** need this if you're using [Khoj Cloud](https://app.khoj.dev).
:::
[Tailscale](https://tailscale.com) simplifies creating a private VPN using [Wireguard](https://www.wireguard.com/) and OAuth. So you can host and access services on your devices from anywhere.
The instructions below are one way to simply and securely access your self-hosted Khoj from your phone, laptop etc.
### Minimal Setup
1. Setup khoj on your preferred machine following the [standard steps](/get-started/setup)
2. Sign-up to [Tailscale](https://tailscale.com) and install the app on machines you want to access Khoj from. This usually includes your khoj server, your phone, laptop. Note the tailscale i.p of your khoj server.
3. Start khoj on your server by including the flag `--host <your_server_tailscale_ip>`
4. Open `http://<your_server_tailscale_ip>:42110` to access khoj from any device on your tailscale network!
### HTTPS Certificate
:::info
Tailscale uses Wireguard to encrypt and route traffic between your machines. So HTTPS isn't required with Tailscale for secure access. HTTPS with Tailscale is only useful for browsers to not complain about security and block certain features like clipboard access unless HTTPS is enabled.
:::
1. Enable [MagicDNS](https://tailscale.com/kb/1081/magicdns#enabling-magicdns) and [HTTPS](https://tailscale.com/kb/1153/enabling-https) toggle on your tailscale admin console [DNS](https://login.tailscale.com/admin/dns) page. Note your unique tailscale domain name (usually ends with .ts.net)
2. Create an https certificate for your Khoj server by running the following command:
```bash
# Assuming the server is named, `server` and your tailnet is `black-forest.ts.net`
# Note path of the .crt and .key files generated
tailscale cert server.black-forest.ts.net
```
3. Start khoj to be served via https on standard port
```bash
sudo KHOJ_DOMAIN=server.black-forest.ts.net \
khoj \
--sslcert /path/to/your/tailscale.crt \
--sslkey path/to/your/tailscale.key \
--host=server.black-forest.ts.net \
--port 443
```
4. You should now be able to access khoj on `https://server.black-forest.ts.net` from any device on your private tailscale network!

13
documentation/docs/advanced/use-openai-proxy.md
View File

@@ -11,7 +11,7 @@ This is only helpful for self-hosted users. If you're using [Khoj Cloud](https:/
Khoj natively supports local LLMs [available on HuggingFace in GGUF format](https://huggingface.co/models?library=gguf). Using an OpenAI API proxy with Khoj maybe useful for ease of setup, trying new models or using commercial LLMs via API.
:::
Khoj can use any OpenAI API compatible server including [Ollama](/advanced/ollama), [LMStudio](/advanced/lmstudio) and [LiteLLM](/advanced/litellm).
Khoj can use any OpenAI API compatible server including local providers like [Ollama](/advanced/ollama), [LMStudio](/advanced/lmstudio) and [LiteLLM](/advanced/litellm) and commercial providers like [HuggingFace](https://huggingface.co/docs/api-inference/tasks/chat-completion#using-the-api), [OpenRouter](https://openrouter.ai/docs/quick-start) etc.
Configuring this allows you to use non-standard, open or commercial, local or hosted LLM models for Khoj
Combine them with Khoj can turn your favorite LLM into an AI agent. Allowing you to chat with your docs, find answers from the internet, build custom agents and run automations.
@@ -20,18 +20,15 @@ For specific integrations, see our [Ollama](/advanced/ollama), [LMStudio](/advan
## General Setup
1. Start your preferred OpenAI API compatible app
2. Create a new [OpenAI Processor Conversation Config](http://localhost:42110/server/admin/database/openaiprocessorconversationconfig/add) on your Khoj admin panel
1. Start your preferred OpenAI API compatible app locally or get API keys from commercial AI model providers.
3. Create a new [API Model API](http://localhost:42110/server/admin/database/aimodelapi/add) on your Khoj admin panel
- Name: `any name`
- Api Key: `any string`
- Api Base Url: **URL of your Openai Proxy API**
3. Create a new [Chat Model Option](http://localhost:42110/server/admin/database/chatmodeloptions/add) on your Khoj admin panel.
3. Create a new [Chat Model](http://localhost:42110/server/admin/database/chatmodel/add) on your Khoj admin panel.
- Name: `llama3` (replace with the name of your local model)
- Model Type: `Openai`
- Openai Config: `<the proxy config you created in step 2>`
- Max prompt size: `2000` (replace with the max prompt size of your model)
- Tokenizer: *Do not set for OpenAI, mistral, llama3 based models*
4. Create a new [Server Chat Setting](http://localhost:42110/server/admin/database/serverchatsettings/add/) on your Khoj admin panel
- Default model: `<name of chat model option you created in step 3>`
- Summarizer model: `<name of chat model option you created in step 3>`
5. Go to [your config](http://localhost:42110/settings) and select the model you just created in the chat model dropdown.
4. Go to [your config](http://localhost:42110/settings) and select the model you just created in the chat model dropdown.

38
documentation/docs/clients/desktop.md
View File

@@ -4,22 +4,13 @@ sidebar_position: 1
# Desktop
> Query your Second Brain from your machine
> Upload your knowledge base to Khoj and chat with your whole corpus
Use the Desktop app to chat and search with Khoj.
You can also share your files, folders with Khoj using the app.
## Companion App
Share your files, folders with Khoj using the app.
Khoj will keep these files in sync to provide contextual responses when you search or chat.
## Features
- **Chat**
- **Faster answers**: Find answers quickly, from your private notes or the public internet
- **Assisted creativity**: Smoothly weave across retrieving answers and generating content
- **Iterative discovery**: Iteratively explore and re-discover your notes
- **Quick access**: Use [Khoj Mini](/features/khoj_mini) on the desktop to quickly pull up a mini chat module for quicker answers
- **Search**
- **Natural**: Advanced natural language understanding using Transformer based ML Models
- **Incremental**: Incremental search for a fast, search-as-you-type experience
## Setup
:::info[Self Hosting]
If you are self-hosting the Khoj server, update the *Settings* page on the Khoj Desktop app to:
@@ -34,7 +25,20 @@ If you are self-hosting the Khoj server, update the *Settings* page on the Khoj
4. [Optional] Add any files, folders you'd like Khoj to be aware of on the *Settings* page and Click *Save*.
These files and folders will be automatically kept in sync for you
## Interface
| Chat | Search |
|:----:|:------:|
| ![](/img/khoj_chat_on_desktop.png) | ![](/img/khoj_search_on_desktop.png) |
# Main App
You can also install the Khoj application on your desktop as a progressive web app.
1. Open the [Khoj Web App](https://app.khoj.dev) in Chrome.
2. Click on the install button in the address bar to install the app. You must be logged into your Chrome browser for this to work.
![progressive web app install icon](/img/pwa_install_desktop.png)
Alternatively, you can also install using this route:
1. Open the three-dot menu in the top right corner of the browser.
2. Go to 'Cast, Save, and Share' option.
3. Click on the "Open in Khoj" option.
![progressive web app install route](/img/chrome_pwa_alt.png)

2
documentation/docs/clients/emacs.md
View File

@@ -114,7 +114,7 @@ This feature finds entries similar to the one you are currently on.
2. Hit `C-c s f` (or `M-x khoj RET f`) to find similar entries
### Advanced Usage
- Add [query filters](https://github.com/khoj-ai/khoj/#query-filters) during search to narrow down results further
- Add [query filters](/miscellaneous/query-filters) during search to narrow down results further
e.g. `What is the meaning of life? -"god" +"none" dt>"last week"`
- Use `C-c C-o 2` to open the current result at cursor in its source org file

2
documentation/docs/clients/obsidian.md
View File

@@ -48,7 +48,7 @@ To see other notes similar to the current one, run *Khoj: Find Similar Notes* fr
### Search
Run *Khoj: Search* from the [Command Palette](https://help.obsidian.md/Plugins/Command+palette)
See [Khoj Search](/features/search) for more details. Use [query filters](/miscellaneous/advanced#query-filters) to limit entries to search
See [Khoj Search](/features/search) for more details. Use [query filters](/miscellaneous/query-filters) to limit entries to search
[search_demo](https://user-images.githubusercontent.com/6413477/218801155-cd67e8b4-a770-404a-8179-d6b61caa0f93.mp4 ':include :type=mp4')

2
documentation/docs/clients/whatsapp.md
View File

@@ -12,7 +12,7 @@ Without any desktop clients, you can start chatting with Khoj on WhatsApp. Bear
In order to use Khoj on WhatsApp with your own data, you need to setup a Khoj Cloud account and connect your WhatsApp account to it. This is a one time setup and you can do it from the [Khoj Cloud config page](https://app.khoj.dev/settings).
If you hit usage limits for the WhatsApp bot, upgrade to [a paid plan](https://khoj.dev/pricing) on Khoj Cloud.
If you hit usage limits for the WhatsApp bot, upgrade to [a paid plan](https://khoj.dev/#pricing) on Khoj Cloud.
<img src="https://khoj-web-bucket.s3.amazonaws.com/khojwhatsapp.png" alt="WhatsApp QR Code" width="300" height="300" />

14
documentation/docs/contributing/development.mdx
View File

@@ -102,7 +102,19 @@ sudo -u postgres createdb khoj --password
</TabItem>
</Tabs>
#### 3. Run
#### 3. Build the front-end assets
```shell
cd src/interface/web/
yarn install
yarn export
```
You can optionally use `yarn dev` to start a development server for the front-end which will be available at http://localhost:3000. This is especially useful if you're making changes to the front-end code, but not necessary for running Khoj. Note that streaming does not work on the dev server due to how it is handled with SSR in Next.js.
Always run `yarn export` to test your front-end changes on http://localhost:42110 before creating a PR.
#### 4. Run
1. Start Khoj
```bash
khoj -vv

2
documentation/docs/data-sources/notion_integration.md
View File

@@ -16,4 +16,4 @@ Go to https://app.khoj.dev/settings to connect your Notion workspace(s) to Khoj.
4. In the first step, you generated an API key. Use the newly generated API Key in your Khoj settings, by default at [http://localhost:42110/settings#notion](http://localhost:42110/settings#notion). Click `Save`.
5. Click `Configure` in http://localhost:42110/settings to index your Notion workspace(s).
That's it! You should be ready to start searching and chatting. Make sure you've configured your [chat settings](/get-started/setup#2-configure).
That's it! You should be ready to start searching and chatting. Make sure you've configured your [chat settings](/get-started/setup#use-khoj).

7
documentation/docs/features/chat.md
View File

@@ -22,7 +22,7 @@ See [the setup guide](/get-started/setup.mdx) to configure your chat models.
- **On Web**: Open [/chat](https://app.khoj.dev/chat) in your web browser
- **On Obsidian**: Search for *Khoj: Chat* in the [Command Palette](https://help.obsidian.md/Plugins/Command+palette)
- **On Emacs**: Run `M-x khoj <user-query>`
2. Enter your queries to chat with Khoj. Use [slash commands](#commands) and [query filters](/miscellaneous/advanced#query-filters) to change what Khoj uses to respond
2. Enter your queries to chat with Khoj. Use [slash commands](#commands) and [query filters](/miscellaneous/query-filters) to change what Khoj uses to respond
#### Details
@@ -30,7 +30,7 @@ See [the setup guide](/get-started/setup.mdx) to configure your chat models.
2. These notes, the last few messages and associated metadata is passed to the enabled chat model along with your query to generate a response
#### Conversation File Filters
You can use conversation file filters to limit the notes used in the chat response. To do so, use the left panel in the web UI. Alternatively, you can also use [query filters](/miscellaneous/advanced#query-filters) to limit the notes used in the chat response.
You can use conversation file filters to limit the notes used in the chat response. To do so, use the left panel in the web UI. Alternatively, you can also use [query filters](/miscellaneous/query-filters) to limit the notes used in the chat response.
<img src="/img/file_filters_conversation.png" alt="Conversation File Filter" style={{width: '400px'}}/>
@@ -43,3 +43,6 @@ Slash commands allows you to change what Khoj uses to respond to your query
- **/image**: Generate an image in response to your query.
- **/help**: Use /help to get all available commands and general information about Khoj
- **/summarize**: Can be used to summarize 1 selected file filter for that conversation. Refer to [File Summarization](summarization) for details.
- **/diagram**: Generate a diagram in response to your query. This is built on [Excalidraw](https://excalidraw.com/).
- **/code**: Generate and run very simple Python code snippets. Refer to [Code Execution](code_execution) for details.
- **/research**: Go deeper in a topic for more accurate, in-depth responses.

30
documentation/docs/features/code_execution.md Normal file
View File

@@ -0,0 +1,30 @@
---
---
# Code Execution
Khoj can generate and run very simple Python code snippets as well. This is useful if you want to generate a plot, run a simple calculation, or do some basic data manipulation. LLMs by default aren't skilled at complex quantitative tasks. Code generation & execution can come in handy for such tasks.
Just use `/code` in your chat command.
### Setup (Self-Hosting)
Run [Cohere's Terrarium](https://github.com/cohere-ai/cohere-terrarium) on your machine to enable code generation and execution.
Check the [instructions](https://github.com/cohere-ai/cohere-terrarium?tab=readme-ov-file#development) for running from source.
For running with Docker, you can use our [docker-compose.yml](https://github.com/khoj-ai/khoj/blob/master/docker-compose.yml), or start it manually like this:
```bash
docker pull ghcr.io/khoj-ai/terrarium:latest
docker run -d -p 8080:8080 ghcr.io/khoj-ai/terrarium:latest
```
#### Verify
Verify that it's running, by evaluating a simple Python expression:
```bash
curl -X POST -H "Content-Type: application/json" \
--url http://localhost:8080 \
--data-raw '{"code": "1 + 1"}' \
--no-buffer
```

17
documentation/docs/features/image_generation.md
View File

@@ -10,6 +10,17 @@ To generate images, you just need to provide a prompt to Khoj in which the image
## Setup (Self-Hosting)
Right now, we only support integration with OpenAI's DALL-E. You need to have an OpenAI API key to use this feature. Here's how you can set it up:
1. Setup your OpenAI API key. See instructions [here](/get-started/setup#2-configure)
2. Create a text to image config at http://localhost:42110/server/admin/database/texttoimagemodelconfig/. We recommend the value `dall-e-3`.
You have a couple of image generation options.
### Image Generation Models
We support most state of the art image generation models, including Ideogram, Flux, and Stable Diffusion. These will run using [Replicate](https://replicate.com). Here's how to set them up:
1. Get a Replicate API key [here](https://replicate.com/account/api-tokens).
2. Create a new [Text to Image Model](http://localhost:42110/server/admin/database/texttoimagemodelconfig/). Set the `type` to `Replicate`. Use any of the model names you see [on this list](https://replicate.com/pricing#image-models). We recommend the `model name` `black-forest-labs/flux-1.1-pro` from [Replicate](https://replicate.com/black-forest-labs/flux-1.1-pro).
### OpenAI
1. Get [an OpenAI API key](https://platform.openai.com/settings/organization/api-keys).
2. Setup your OpenAI API key, if you haven't already. See instructions [here](/get-started/setup#add-chat-models)
3. Create a text to image config at http://localhost:42110/server/admin/database/texttoimagemodelconfig/. Use `model name` `dall-e-3` to use openai for image generation. Make sure to set the `Ai model api` field to the the OpenAI AI model api you setup in step 2.

16
documentation/docs/features/online_search.md
View File

@@ -14,8 +14,18 @@ Try it out yourself! https://app.khoj.dev
## Self-Hosting
Online search works out of the box even when self-hosting. Khoj uses [JinaAI's reader API](https://jina.ai/reader/) to search online and read webpages by default. No API key setup is necessary.
### Search
To improve online search, set the `SERPER_DEV_API_KEY` environment variable to your [Serper.dev](https://serper.dev/) API key. These search results include additional context like answer box, knowledge graph etc.
Online search can work even with self-hosting! You have a few options:
For advanced webpage reading, set the `OLOSTEP_API_KEY` environment variable to your [Olostep](https://www.olostep.com/) API key. This has a higher success rate at reading webpages than the default webpage reader.
- If you're using Docker, online search should work out of the box with [searxng](https://github.com/searxng/searxng) using our standard `docker-compose.yml`.
- For a non-local, free solution, you can use [JinaAI's reader API](https://jina.ai/reader/) to search online and read webpages. You can get a free API key via https://jina.ai/reader. Set the `JINA_API_KEY` environment variable to your Jina AI reader API key to enable online search.
- To get production-grade, fast online search, set the `SERPER_DEV_API_KEY` environment variable to your [Serper.dev](https://serper.dev/) API key. These search results include additional context like answer box, knowledge graph etc.
### Webpage Reading
Out of the box, you **don't have to do anything to enable webpage reading**. Khoj will automatically read webpages by using the `requests` library. To get more distributed and scalable webpage reading, you can use the following options:
- If you're using Jina AI's reader API for search, it should work automatically for webpage reading as well.
- For scalable webpage scraping, you can use [Firecrawl](https://www.firecrawl.dev/). Create a new [Webscraper](http://localhost:42110/server/admin/database/webscraper/add/). Set your Firecrawl API key to the Api Key field, and set the type to Firecrawl.
- For advanced webpage reading, you can use [Olostep](https://www.olostep.com/). This has a higher success rate at reading webpages than the default webpage readers. Create a new [Webscraper](http://localhost:42110/server/admin/database/webscraper/add/). Set your Olostep API key to the Api Key field, and set the type to Olostep.

2
documentation/docs/features/search.md
View File

@@ -11,7 +11,7 @@ Take advantage of super fast search to find relevant notes and documents from yo
- **On Web**: Open https://app.khoj.dev/ in your web browser
- **On Obsidian**: Click the *Khoj search* icon 🔎 on the [Ribbon](https://help.obsidian.md/User+interface/Workspace/Ribbon) or Search for *Khoj: Search* in the [Command Palette](https://help.obsidian.md/Plugins/Command+palette)
- **On Emacs**: Run `M-x khoj <user-query>`
2. Query using natural language to find relevant entries from your knowledge base. Use [query filters](/miscellaneous/advanced#query-filters) to limit entries to search
2. Query using natural language to find relevant entries from your knowledge base. Use [query filters](/miscellaneous/query-filters) to limit entries to search
### Demo
![](/img/search_agents_markdown.png ':size=400px')

4
documentation/docs/features/voice-chat.md
View File

@@ -17,11 +17,11 @@ You can also click on the speaker icon next to any message to hear it out loud.
Voice chat will automatically be configured when you initialize the application. The default configuration will run locally. If you want to use the OpenAI whisper API for voice chat, you can set it up by following these steps:
1. Setup your OpenAI API key. See instructions [here](/get-started/setup#2-configure).
1. Setup your OpenAI API key. See instructions [here](/get-started/setup#add-chat-models).
2. Create a new configuration at http://localhost:42110/server/admin/database/speechtotextmodeloptions/. We recommend the value `whisper-1` and model type `Openai`.
If you want to use the Text to Speech feature, you can set it up by following these steps:
1. Setup your account on [ElevenLabs.io](https://elevenlabs.io/).
2. Configure your API key in your environment variables with the key `ELEVEN_LABS_API_KEY`.
2. (Optional) Create a new [Voice model option](http://localhost:42110/server/admin/database/voicemodeloption/) with a specific voice ID from whichever voice you want to use. You can explore the options [here](https://elevenlabs.io/app/voice-library).
3. (Optional) Create a new [Voice model option](http://localhost:42110/server/admin/database/voicemodeloption/) with a specific voice ID from whichever voice you want to use. You can explore the options [here](https://elevenlabs.io/app/voice-library).

5
documentation/docs/get-started/overview.md
View File

@@ -1,13 +1,10 @@
---
sidebar_position: 0
slug: /
keywords: ["khoj", "khoj ai", "khoj docs", "khoj documentation", "khoj features", "khoj overview", "khoj quickstart", "khoj chat", "khoj search", "khoj cloud", "khoj self-host", "khoj setup", "open source ai", "local llm", "ai copilot", "second brain", "personal ai", "ai search engine"]
keywords: ["khoj", "khoj ai", "khoj docs", "khoj documentation", "khoj features", "khoj overview", "khoj quickstart", "khoj chat", "khoj search", "khoj cloud", "khoj self-host", "khoj setup", "open source ai", "local llm", "ai copilot", "second brain", "personal ai", "ai search engine", "research assistant", "open source llm", "self-host llm", "self-host chatgpt", "ai chatbot", "ai assistant", "ai research assistant", "ai search engine", "ai knowledge base", "ai knowledge graph", "ai personal assistant", "ai second brain", "open source ai assistant"]
---
# Overview
<p align="center"><img src="/img/khoj-logo-sideways-500.png" width="200" alt="Khoj Logo"></img></p>
<div align="center">
<b>Your Second Brain</b>
</div>

103
documentation/docs/get-started/setup.mdx
View File

@@ -19,7 +19,11 @@ These are the general setup instructions for self-hosted Khoj.
You can install the Khoj server using either [Docker](?server=docker) or [Pip](?server=pip).
:::info[Offline Model + GPU]
If you want to use the offline chat model and you have a GPU, you should use Installation Option 2 - local setup via the Python package directly. Our Docker image doesn't currently support running the offline chat model on GPU, making inference times really slow.
To use the offline chat model with your GPU, we recommend using the Docker setup with Ollama . You can also use the local Khoj setup via the Python package directly.
:::
:::info[First Run]
Restart your Khoj server after the first run to ensure all settings are applied correctly.
:::
<Tabs groupId="server" queryString>
@@ -27,7 +31,29 @@ If you want to use the offline chat model and you have a GPU, you should use Ins
<Tabs groupId="operating-systems" queryString="os">
<TabItem value="macos" label="MacOS">
<h3>Prerequisites</h3>
Install [Docker Desktop](https://docs.docker.com/desktop/install/mac-install/)
<h4>Docker</h4>
- *Option 1*: Click here to install [Docker Desktop](https://docs.docker.com/desktop/install/mac-install/). Make sure you also install the [Docker Compose](https://docs.docker.com/desktop/install/mac-install/) tool.
- *Option 2*: Use [Homebrew](https://brew.sh/) to install Docker and Docker Compose.
```shell
brew install --cask docker
brew install docker-compose
```
<h3>Setup</h3>
1. Download the Khoj docker-compose.yml file [from Github](https://github.com/khoj-ai/khoj/blob/master/docker-compose.yml)
```shell
mkdir ~/.khoj && cd ~/.khoj
wget https://raw.githubusercontent.com/khoj-ai/khoj/master/docker-compose.yml
```
2. Configure the environment variables in the `docker-compose.yml`
- Set `KHOJ_ADMIN_PASSWORD`, `KHOJ_DJANGO_SECRET_KEY` (and optionally the `KHOJ_ADMIN_EMAIL`) to something secure. This allows you to customize Khoj later via the admin panel.
- Set `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, or `GEMINI_API_KEY` to your API key if you want to use OpenAI, Anthropic or Gemini commercial chat models respectively.
- Uncomment `OPENAI_API_BASE` to use [Ollama](/advanced/ollama?type=first-run&server=docker#setup) running on your host machine. Or set it to the URL of your OpenAI compatible API like vLLM or [LMStudio](/advanced/lmstudio).
3. Start Khoj by running the following command in the same directory as your docker-compose.yml file.
```shell
cd ~/.khoj
docker-compose up
```
</TabItem>
<TabItem value="windows" label="Windows">
<h3>Prerequisites</h3>
@@ -37,30 +63,49 @@ If you want to use the offline chat model and you have a GPU, you should use Ins
wsl --install
```
2. Install [Docker Desktop](https://docs.docker.com/desktop/install/windows-install/) with **[WSL2 backend](https://docs.docker.com/desktop/wsl/#turn-on-docker-desktop-wsl-2)** (default)
<h3>Setup</h3>
1. Download the Khoj docker-compose.yml file [from Github](https://github.com/khoj-ai/khoj/blob/master/docker-compose.yml)
```shell
# Windows users should use their WSL2 terminal to run these commands
mkdir ~/.khoj && cd ~/.khoj
wget https://raw.githubusercontent.com/khoj-ai/khoj/master/docker-compose.yml
```
2. Configure the environment variables in the `docker-compose.yml`
- Set `KHOJ_ADMIN_PASSWORD`, `KHOJ_DJANGO_SECRET_KEY` (and optionally the `KHOJ_ADMIN_EMAIL`) to something secure. This allows you to customize Khoj later via the admin panel.
- Set `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, or `GEMINI_API_KEY` to your API key if you want to use OpenAI, Anthropic or Gemini commercial chat models respectively.
- Uncomment `OPENAI_API_BASE` to use [Ollama](/advanced/ollama) running on your host machine. Or set it to the URL of your OpenAI compatible API like vLLM or [LMStudio](/advanced/lmstudio).
3. Start Khoj by running the following command in the same directory as your docker-compose.yml file.
```shell
# Windows users should use their WSL2 terminal to run these commands
cd ~/.khoj
docker-compose up
```
</TabItem>
<TabItem value="linux" label="Linux">
<h3>Prerequisites</h3>
Install [Docker Desktop](https://docs.docker.com/desktop/install/windows-install/).
Install [Docker Desktop](https://docs.docker.com/desktop/install/linux/).
You can also use your package manager to install Docker Engine & Docker Compose.
<h3>Setup</h3>
1. Download the Khoj docker-compose.yml file [from Github](https://github.com/khoj-ai/khoj/blob/master/docker-compose.yml)
```shell
mkdir ~/.khoj && cd ~/.khoj
wget https://raw.githubusercontent.com/khoj-ai/khoj/master/docker-compose.yml
```
2. Configure the environment variables in the `docker-compose.yml`
- Set `KHOJ_ADMIN_PASSWORD`, `KHOJ_DJANGO_SECRET_KEY` (and optionally the `KHOJ_ADMIN_EMAIL`) to something secure. This allows you to customize Khoj later via the admin panel.
- Set `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, or `GEMINI_API_KEY` to your API key if you want to use OpenAI, Anthropic or Gemini commercial chat models respectively.
- Uncomment `OPENAI_API_BASE` to use [Ollama](/advanced/ollama) running on your host machine. Or set it to the URL of your OpenAI compatible API like vLLM or [LMStudio](/advanced/lmstudio).
3. Start Khoj by running the following command in the same directory as your docker-compose.yml file.
```shell
cd ~/.khoj
docker-compose up
```
</TabItem>
</Tabs>
<h3>Setup</h3>
1. Download the Khoj docker-compose.yml file [from Github](https://github.com/khoj-ai/khoj/blob/master/docker-compose.yml)
```shell
# Windows users should use their WSL2 terminal to run these commands
mkdir ~/.khoj && cd ~/.khoj
wget https://raw.githubusercontent.com/khoj-ai/khoj/master/docker-compose.yml
```
2. Configure the environment variables in the docker-compose.yml
- Set `KHOJ_ADMIN_PASSWORD`, `KHOJ_DJANGO_SECRET_KEY` (and optionally the `KHOJ_ADMIN_EMAIL`) to something secure. This allows you to customize Khoj later via the admin panel.
- Set `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, or `GEMINI_API_KEY` to your API key if you want to use OpenAI, Anthropic or Gemini chat models respectively.
3. Start Khoj by running the following command in the same directory as your docker-compose.yml file.
```shell
# Windows users should use their WSL2 terminal to run these commands
cd ~/.khoj
docker-compose up
```
:::info[Remote Access]
By default Khoj is only accessible on the machine it is running. To access Khoj from a remote machine see [Remote Access Docs](/advanced/remote).
@@ -257,12 +302,12 @@ Setup which chat model you'd want to use. Khoj supports local and online chat mo
Using Ollama? See the [Ollama Integration](/advanced/ollama) section for more custom setup instructions.
:::
1. Create a new [OpenAI processor conversation config](http://localhost:42110/server/admin/database/openaiprocessorconversationconfig/add) in the server admin settings. This is kind of a misnomer, we know.
1. Create a new [AI Model Api](http://localhost:42110/server/admin/database/aimodelapi/add) in the server admin settings.
- Add your [OpenAI API key](https://platform.openai.com/api-keys)
- Give the configuration a friendly name like `OpenAI`
- (Optional) Set the API base URL. It is only relevant if you're using another OpenAI-compatible proxy server like [Ollama](/advanced/ollama) or [LMStudio](/advanced/lmstudio).<br />
![example configuration for openai processor](/img/example_openai_processor_config.png)
2. Create a new [chat model options](http://localhost:42110/server/admin/database/chatmodeloptions/add)
![example configuration for ai model api](/img/example_openai_processor_config.png)
2. Create a new [chat model](http://localhost:42110/server/admin/database/chatmodel/add)
- Set the `chat-model` field to an [OpenAI chat model](https://platform.openai.com/docs/models). Example: `gpt-4o`.
- Make sure to set the `model-type` field to `OpenAI`.
- If your model supports vision, set the `vision enabled` field to `true`. This is currently only supported for OpenAI models with vision capabilities.
@@ -270,22 +315,22 @@ Using Ollama? See the [Ollama Integration](/advanced/ollama) section for more cu
![example configuration for chat model options](/img/example_chatmodel_option.png)
</TabItem>
<TabItem value="anthropic" label="Anthropic">
1. Create a new [OpenAI processor conversation config](http://localhost:42110/server/admin/database/openaiprocessorconversationconfig/add) in the server admin settings. This is kind of a misnomer, we know.
1. Create a new [AI Model API](http://localhost:42110/server/admin/database/aimodelapi/add) in the server admin settings.
- Add your [Anthropic API key](https://console.anthropic.com/account/keys)
- Give the configuration a friendly name like `Anthropic`. Do not configure the API base url.
2. Create a new [chat model options](http://localhost:42110/server/admin/database/chatmodeloptions/add)
2. Create a new [chat model](http://localhost:42110/server/admin/database/chatmodel/add)
- Set the `chat-model` field to an [Anthropic chat model](https://docs.anthropic.com/en/docs/about-claude/models#model-names). Example: `claude-3-5-sonnet-20240620`.
- Set the `model-type` field to `Anthropic`.
- Set the `Openai config` field to the OpenAI processor conversation config for Anthropic you created in step 1.
- Set the `ai model api` field to the Anthropic AI Model API you created in step 1.
</TabItem>
<TabItem value="gemini" label="Gemini">
1. Create a new [OpenAI processor conversation config](http://localhost:42110/server/admin/database/openaiprocessorconversationconfig/add) in the server admin settings. This is kind of a misnomer, we know.
1. Create a new [AI Model API](http://localhost:42110/server/admin/database/aimodelapi/add) in the server admin settings.
- Add your [Gemini API key](https://aistudio.google.com/app/apikey)
- Give the configuration a friendly name like `Gemini`. Do not configure the API base url.
2. Create a new [chat model options](http://localhost:42110/server/admin/database/chatmodeloptions/add)
2. Create a new [chat model](http://localhost:42110/server/admin/database/chatmodel/add)
- Set the `chat-model` field to a [Google Gemini chat model](https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#gemini-models). Example: `gemini-1.5-flash`.
- Set the `model-type` field to `Gemini`.
- Set the `Openai config` field to the OpenAI processor conversation config for Gemini you created in step 1.
- Set the `ai model api` field to the Gemini AI Model API you created in step 1.
</TabItem>
<TabItem value="offline" label="Offline">
@@ -298,7 +343,7 @@ Offline chat stays completely private and can work without internet using any op
:::
1. Get the name of your preferred chat model from [HuggingFace](https://huggingface.co/models?pipeline_tag=text-generation&library=gguf). *Most GGUF format chat models are supported*.
2. Open the [create chat model page](http://localhost:42110/server/admin/database/chatmodeloptions/add/) on the admin panel
2. Open the [create chat model page](http://localhost:42110/server/admin/database/chatmodel/add/) on the admin panel
3. Set the `chat-model` field to the name of your preferred chat model
- Make sure the `model-type` is set to `Offline`
4. Set the newly added chat model as your preferred model in your [User chat settings](http://localhost:42110/settings) and [Server chat settings](http://localhost:42110/server/admin/database/serverchatsettings/).

10
documentation/docs/miscellaneous/performance.md
View File

@@ -6,20 +6,22 @@ sidebar_position: 2
Here are some top-level performance metrics for Khoj. These are rough estimates and will vary based on your hardware and data.
:::info
These performance metrics were last evaluated in 2022.
:::
### Search performance
- Semantic search using the bi-encoder is fairly fast at \<100 ms across all content types
- Reranking using the cross-encoder is slower at \<2s on 15 results. Tweak `top_k` to tradeoff speed for accuracy of results
- Semantic search using the default embeddings model is fairly fast at \<100 ms across all content types
- Reranking using the cross-encoder model is slower at \<2s on 15 results. Tweak `top_k` to tradeoff speed for accuracy of results
- Filters in query (e.g. by file, word or date) usually add \<20ms to query latency
### Indexing performance
- Indexing is more strongly impacted by the size of the source data
- Indexing 100K+ line corpus of notes takes about 10 minutes
- Indexing 4000+ images takes about 15 minutes and more than 8Gb of RAM
- Note: *It should only take this long on the first run* as the index is incrementally updated
### Miscellaneous
- Testing done on a Mac M1 and a \>100K line corpus of notes
- Search, indexing on a GPU has not been tested yet

4
documentation/docs/miscellaneous/advanced.md → documentation/docs/miscellaneous/query-filters.md
View File

@@ -2,9 +2,7 @@
sidebar_position: 3
---
# Advanced Usage
## Query Filters
# Query Filters
Use structured query syntax to filter entries from your knowledge based used by search results or chat responses.

9
documentation/docs/miscellaneous/telemetry.md
View File

@@ -14,13 +14,6 @@ We don't send any personal information or any information from/about your conten
## Disable Telemetry
If you're self-hosting Khoj, you can opt out of telemetry at any time. To do so,
1. Open `~/.khoj/khoj.yml`
2. Add the following configuration:
```
app:
should-log-telemetry: false
```
3. Save the file and restart Khoj
If you're self-hosting Khoj, you can opt out of telemetry at any time by setting the `KHOJ_TELEMETRY_DISABLE` environment variable to `True` via shell or `docker-compose.yml`
If you have any questions or concerns, please reach out to us on [Discord](https://discord.gg/BDgyabRM6e).

27
documentation/docusaurus.config.js
View File

@@ -37,6 +37,9 @@ const config = {
locales: ['en'],
},
// Add a widget for Chatwoot for live chat if users need help
clientModules: [require.resolve('./src/components/ChatwootWidget.js')],
presets: [
[
'classic',
@@ -69,19 +72,17 @@ const config = {
}),
],
],
themeConfig:
/** @type {import('@docusaurus/preset-classic').ThemeConfig} */
({
image: 'img/khoj-logo-sideways-500.png',
image: 'img/khoj_documentation.png',
metadata: [
{name: 'og:title', content: 'Khoj Documentation'},
{name: 'og:title', content: 'Docs'},
{name: 'og:type', content: 'website'},
{name: 'og:site_name', content: 'Khoj Documentation'},
{name: 'og:description', content: 'Quickly get started with using or self-hosting Khoj'},
{name: 'og:image', content: 'https://assets.khoj.dev/link_preview_docs.png'},
{name: 'og:url', content: 'https://docs.khoj.dev'},
{name: 'keywords', content: 'khoj, khoj ai, chatgpt, open ai, open source, productivity'}
{name: 'keywords', content: 'khoj, khoj ai, chatgpt, open source ai, open source, transparent, accessible, trustworthy, hackable, index notes, rag, productivity'}
],
navbar: {
title: 'Khoj',
@@ -92,23 +93,31 @@ const config = {
items: [
{
href: 'https://github.com/khoj-ai/khoj',
label: 'GitHub',
position: 'right',
className: 'header-github-link',
title: 'Codebase',
'aria-label': 'GitHub repository',
},
{
href: 'https://app.khoj.dev/login',
label: 'Cloud',
position: 'right',
className: 'header-cloud-link',
title: 'Khoj Cloud',
'aria-label': 'Khoj Cloud',
},
{
href: 'https://discord.gg/BDgyabRM6e',
label: 'Discord',
position: 'right',
className: 'header-discord-link',
title: 'Community',
'aria-label': 'Discord community',
},
{
href: 'https://blog.khoj.dev',
label: 'Blog',
position: 'right',
className: 'header-blog-link',
title: 'Blog',
'aria-label': 'Khoj Blog',
},
],
},

19
documentation/src/components/ChatwootWidget.js Normal file
View File

@@ -0,0 +1,19 @@
import ExecutionEnvironment from '@docusaurus/ExecutionEnvironment';
// Only execute on client-side
if (ExecutionEnvironment.canUseDOM) {
(function (d, t) {
var BASE_URL = "https://app.chatwoot.com";
var g = d.createElement(t), s = d.getElementsByTagName(t)[0];
g.src = BASE_URL + "/packs/js/sdk.js";
g.defer = true;
g.async = true;
s.parentNode.insertBefore(g, s);
g.onload = function () {
window.chatwootSDK.run({
websiteToken: 'cFxvnLSjfE2UF4UUiPCA5NsF',
baseUrl: BASE_URL
})
}
})(document, 'script');
}

76
documentation/src/css/custom.css
View File

@@ -8,13 +8,13 @@
/* You can override the default Infima variables here. */
:root {
--ifm-color-primary: #fcc50b;
--ifm-color-primary-dark: #fcc50b;
--ifm-color-primary-darker: #fcc50b;
--ifm-color-primary-darkest: #fcc50b;
--ifm-color-primary-light: #fcc50b;
--ifm-color-primary-lighter: #fcc50b;
--ifm-color-primary-lightest: #fcc50b;
--ifm-color-primary: #FFA07A;
--ifm-color-primary-dark: #FFA07A;
--ifm-color-primary-darker: #FFA07A;
--ifm-color-primary-darkest: #FFA07A;
--ifm-color-primary-light: #FFA07A;
--ifm-color-primary-lighter: #FFA07A;
--ifm-color-primary-lightest: #FFA07A;
--ifm-code-font-size: 95%;
--ifm-heading-font-family: 'Source Sans 3', sans-serif;
--docusaurus-highlighted-code-line-bg: rgba(0, 0, 0, 0.1);
@@ -22,16 +22,64 @@
/* For readability concerns, you should choose a lighter palette in dark mode. */
[data-theme='dark'] {
--ifm-color-primary: #fcc50b;
--ifm-color-primary-dark: #fcc50b;
--ifm-color-primary-darker: #fcc50b;
--ifm-color-primary-darkest: #fcc50b;
--ifm-color-primary-light: #fcc50b;
--ifm-color-primary-lighter: #fcc50b;
--ifm-color-primary-lightest: #fcc50b;
--ifm-color-primary: #FFA07A;
--ifm-color-primary-dark: #FFA07A;
--ifm-color-primary-darker: #FFA07A;
--ifm-color-primary-darkest: #FFA07A;
--ifm-color-primary-light: #FFA07A;
--ifm-color-primary-lighter: #FFA07A;
--ifm-color-primary-lightest: #FFA07A;
--docusaurus-highlighted-code-line-bg: rgba(0, 0, 0, 0.3);
}
body {
font-family: 'Source Sans 3', sans-serif;
}
.header-github-link::before {
content: '';
width: 24px;
height: 24px;
display: flex;
background-color: var(--ifm-navbar-link-color);
mask-image: url("data:image/svg+xml,%3Csvg viewBox='0 0 24 24' xmlns='http://www.w3.org/2000/svg'%3E%3Cpath d='M12 .297c-6.63 0-12 5.373-12 12 0 5.303 3.438 9.8 8.205 11.385.6.113.82-.258.82-.577 0-.285-.01-1.04-.015-2.04-3.338.724-4.042-1.61-4.042-1.61C4.422 18.07 3.633 17.7 3.633 17.7c-1.087-.744.084-.729.084-.729 1.205.084 1.838 1.236 1.838 1.236 1.07 1.835 2.809 1.305 3.495.998.108-.776.417-1.305.76-1.605-2.665-.3-5.466-1.332-5.466-5.93 0-1.31.465-2.38 1.235-3.22-.135-.303-.54-1.523.105-3.176 0 0 1.005-.322 3.3 1.23.96-.267 1.98-.399 3-.405 1.02.006 2.04.138 3 .405 2.28-1.552 3.285-1.23 3.285-1.23.645 1.653.24 2.873.12 3.176.765.84 1.23 1.91 1.23 3.22 0 4.61-2.805 5.625-5.475 5.92.42.36.81 1.096.81 2.22 0 1.606-.015 2.896-.015 3.286 0 .315.21.69.825.57C20.565 22.092 24 17.592 24 12.297c0-6.627-5.373-12-12-12'/%3E%3C/svg%3E");
transition: background-color var(--ifm-transition-fast) var(--ifm-transition-timing-default);
}
.header-blog-link::before {
content: '';
width: 24px;
height: 24px;
display: flex;
background-color: var(--ifm-navbar-link-color);
mask-image: url("data:image/svg+xml,%3Csvg viewBox='0 0 512 512' xmlns='http://www.w3.org/2000/svg'%3E%3C!--!Font Awesome Free 6.7.2 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free Copyright 2024 Fonticons, Inc.--%3E%3Cpath d='M278.5 215.6L23 471c-9.4 9.4-9.4 24.6 0 33.9s24.6 9.4 33.9 0l57-57 68 0c49.7 0 97.9-14.4 139-41c11.1-7.2 5.5-23-7.8-23c-5.1 0-9.2-4.1-9.2-9.2c0-4.1 2.7-7.6 6.5-8.8l81-24.3c2.5-.8 4.8-2.1 6.7-4l22.4-22.4c10.1-10.1 2.9-27.3-11.3-27.3l-32.2 0c-5.1 0-9.2-4.1-9.2-9.2c0-4.1 2.7-7.6 6.5-8.8l112-33.6c4-1.2 7.4-3.9 9.3-7.7C506.4 207.6 512 184.1 512 160c0-41-16.3-80.3-45.3-109.3l-5.5-5.5C432.3 16.3 393 0 352 0s-80.3 16.3-109.3 45.3L139 149C91 197 64 262.1 64 330l0 55.3L253.6 195.8c6.2-6.2 16.4-6.2 22.6 0c5.4 5.4 6.1 13.6 2.2 19.8z'/%3E%3C/svg%3E");
transition: background-color var(--ifm-transition-fast) var(--ifm-transition-timing-default);
}
.header-cloud-link::before {
content: '';
width: 24px;
height: 24px;
display: flex;
background-color: var(--ifm-navbar-link-color);
mask-image: url("data:image/svg+xml,%3Csvg viewBox='0 0 512 512' xmlns='http://www.w3.org/2000/svg'%3E%3C!--!Font Awesome Free 6.7.2 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free Copyright 2024 Fonticons, Inc.--%3E%3Cpath d='M0 224c0 53 43 96 96 96l47.2 0L290 202.5c17.6-14.1 42.6-14 60.2 .2s22.8 38.6 12.8 58.8L333.7 320l18.3 0 64 0c53 0 96-43 96-96s-43-96-96-96c-.5 0-1.1 0-1.6 0c1.1-5.2 1.6-10.5 1.6-16c0-44.2-35.8-80-80-80c-24.3 0-46.1 10.9-60.8 28C256.5 24.3 219.1 0 176 0C114.1 0 64 50.1 64 112c0 7.1 .7 14.1 1.9 20.8C27.6 145.4 0 181.5 0 224zm330.1 3.6c-5.8-4.7-14.2-4.7-20.1-.1l-160 128c-5.3 4.2-7.4 11.4-5.1 17.8s8.3 10.7 15.1 10.7l70.1 0L177.7 488.8c-3.4 6.7-1.6 14.9 4.3 19.6s14.2 4.7 20.1 .1l160-128c5.3-4.2 7.4-11.4 5.1-17.8s-8.3-10.7-15.1-10.7l-70.1 0 52.4-104.8c3.4-6.7 1.6-14.9-4.2-19.6z'/%3E%3C/svg%3E");
transition: background-color var(--ifm-transition-fast) var(--ifm-transition-timing-default);
}
.header-discord-link::before {
content: '';
width: 30px;
height: 24px;
display: flex;
background-color: var(--ifm-navbar-link-color);
mask-image: url("data:image/svg+xml,%3Csvg viewBox='0 0 640 512' xmlns='http://www.w3.org/2000/svg'%3E%3C!--!Font Awesome Free 6.7.2 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free Copyright 2024 Fonticons, Inc.--%3E%3Cpath d='M524.5 69.8a1.5 1.5 0 0 0 -.8-.7A485.1 485.1 0 0 0 404.1 32a1.8 1.8 0 0 0 -1.9 .9 337.5 337.5 0 0 0 -14.9 30.6 447.8 447.8 0 0 0 -134.4 0 309.5 309.5 0 0 0 -15.1-30.6 1.9 1.9 0 0 0 -1.9-.9A483.7 483.7 0 0 0 116.1 69.1a1.7 1.7 0 0 0 -.8 .7C39.1 183.7 18.2 294.7 28.4 404.4a2 2 0 0 0 .8 1.4A487.7 487.7 0 0 0 176 479.9a1.9 1.9 0 0 0 2.1-.7A348.2 348.2 0 0 0 208.1 430.4a1.9 1.9 0 0 0 -1-2.6 321.2 321.2 0 0 1 -45.9-21.9 1.9 1.9 0 0 1 -.2-3.1c3.1-2.3 6.2-4.7 9.1-7.1a1.8 1.8 0 0 1 1.9-.3c96.2 43.9 200.4 43.9 295.5 0a1.8 1.8 0 0 1 1.9 .2c2.9 2.4 6 4.9 9.1 7.2a1.9 1.9 0 0 1 -.2 3.1 301.4 301.4 0 0 1 -45.9 21.8 1.9 1.9 0 0 0 -1 2.6 391.1 391.1 0 0 0 30 48.8 1.9 1.9 0 0 0 2.1 .7A486 486 0 0 0 610.7 405.7a1.9 1.9 0 0 0 .8-1.4C623.7 277.6 590.9 167.5 524.5 69.8zM222.5 337.6c-29 0-52.8-26.6-52.8-59.2S193.1 219.1 222.5 219.1c29.7 0 53.3 26.8 52.8 59.2C275.3 311 251.9 337.6 222.5 337.6zm195.4 0c-29 0-52.8-26.6-52.8-59.2S388.4 219.1 417.9 219.1c29.7 0 53.3 26.8 52.8 59.2C470.7 311 447.5 337.6 417.9 337.6z'/%3E%3C/svg%3E");
transition: background-color var(--ifm-transition-fast) var(--ifm-transition-timing-default);
}
.header-github-link:hover::before,
.header-cloud-link:hover::before,
.header-discord-link:hover::before,
.header-blog-link:hover::before {
background-color: var(--ifm-navbar-link-hover-color);
transform: scale(1.1);
}

3658
documentation/yarn.lock
View File

File diff suppressed because it is too large Load Diff

2
manifest.json
View File

@@ -1,7 +1,7 @@
{
"id": "khoj",
"name": "Khoj",
"version": "1.26.2",
"version": "1.33.1",
"minAppVersion": "0.15.0",
"description": "Your Second Brain",
"author": "Khoj Inc.",

50
prod.Dockerfile
View File

@@ -1,5 +1,5 @@
# syntax=docker/dockerfile:1
FROM ubuntu:jammy
FROM ubuntu:jammy AS base
LABEL homepage="https://khoj.dev"
LABEL repository="https://github.com/khoj-ai/khoj"
LABEL org.opencontainers.image.source="https://github.com/khoj-ai/khoj"
@@ -13,36 +13,52 @@ RUN apt update -y && apt -y install \
libsm6 \
libxext6 \
swig \
curl && \
# Required by Next.js Web app
curl -sL https://deb.nodesource.com/setup_20.x | bash - && \
curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add - && \
echo "deb https://dl.yarnpkg.com/debian/ stable main" | tee /etc/apt/sources.list.d/yarn.list && \
apt update -y && apt -y --no-install-recommends install nodejs yarn && \
curl \
# Required by llama-cpp-python pre-built wheels. See #1628
musl-dev && \
ln -s /usr/lib/x86_64-linux-musl/libc.so /lib/libc.musl-x86_64.so.1 && \
# Clean up
apt clean && rm -rf /var/lib/apt/lists/*
# Install Application
# Build Server
FROM base AS server-deps
WORKDIR /app
COPY pyproject.toml .
COPY README.md .
ARG VERSION=0.0.0
# use the pre-built llama-cpp-python, torch cpu wheel
ENV PIP_EXTRA_INDEX_URL="https://download.pytorch.org/whl/cpu https://abetlen.github.io/llama-cpp-python/whl/cpu"
# avoid downloading unused cuda specific python packages
ENV CUDA_VISIBLE_DEVICES=""
RUN sed -i "s/dynamic = \\[\"version\"\\]/version = \"$VERSION\"/" pyproject.toml && \
pip install --no-cache-dir -e .[prod]
# Copy Source Code
COPY . .
# Set the PYTHONPATH environment variable in order for it to find the Django app.
ENV PYTHONPATH=/app/src:$PYTHONPATH
# Go to the directory src/interface/web and export the built Next.js assets
# Build Web App
FROM node:20-alpine AS web-app
# Set build optimization env vars
ENV NODE_ENV=production
ENV NEXT_TELEMETRY_DISABLED=1
WORKDIR /app/src/interface/web
RUN bash -c "yarn install --frozen-lockfile --verbose && yarn ciexport && yarn cache clean"
# Install dependencies first (cache layer)
COPY src/interface/web/package.json src/interface/web/yarn.lock ./
RUN yarn install --frozen-lockfile
# Copy source and build
COPY src/interface/web/. ./
RUN yarn build
# Merge the Server and Web App into a Single Image
FROM base
ENV PYTHONPATH=/app/src:$PYTHONPATH
WORKDIR /app
COPY --from=server-deps /usr/local/lib/python3.10/dist-packages /usr/local/lib/python3.10/dist-packages
COPY --from=server-deps /usr/local/bin /usr/local/bin
COPY --from=web-app /app/src/interface/web/out ./src/khoj/interface/built
COPY . .
RUN cd src && python3 khoj/manage.py collectstatic --noinput
# Run the Application
# There are more arguments required for the application to run,
# but these should be passed in through the docker-compose.yml file.
# but those should be passed in through the docker-compose.yml file.
ARG PORT
EXPOSE ${PORT}
ENTRYPOINT ["gunicorn", "-c", "gunicorn-config.py", "src.khoj.main:app"]

15
pyproject.toml
View File

@@ -41,7 +41,7 @@ dependencies = [
"defusedxml == 0.7.1",
"fastapi >= 0.110.0",
"python-multipart >= 0.0.7",
"jinja2 == 3.1.4",
"jinja2 == 3.1.5",
"openai >= 1.0.0",
"tiktoken >= 0.3.2",
"tenacity >= 8.2.2",
@@ -62,8 +62,9 @@ dependencies = [
"requests >= 2.26.0",
"tenacity == 8.3.0",
"anyio == 3.7.1",
"pymupdf >= 1.23.5",
"django == 5.0.9",
"pymupdf == 1.24.11",
"django == 5.0.10",
"django-unfold == 0.42.0",
"authlib == 1.2.1",
"llama-cpp-python == 0.2.88",
"itsdangerous == 2.1.2",
@@ -87,7 +88,9 @@ dependencies = [
"django_apscheduler == 0.6.2",
"anthropic == 0.26.1",
"docx2txt == 0.8",
"google-generativeai == 0.7.2"
"google-generativeai == 0.8.3",
"pyjson5 == 1.6.7",
"resend == 1.0.1",
]
dynamic = ["version"]
@@ -106,7 +109,6 @@ prod = [
"stripe == 7.3.0",
"twilio == 8.11",
"boto3 >= 1.34.57",
"resend == 1.0.1",
]
dev = [
"khoj[prod]",
@@ -119,6 +121,9 @@ dev = [
"mypy >= 1.0.1",
"black >= 23.1.0",
"pre-commit >= 3.0.4",
"gitpython ~= 3.1.43",
"datasets",
"pandas",
]
[tool.hatch.version]

211
src/interface/android/app/build.gradle Normal file
View File

@@ -0,0 +1,211 @@
/*
* Copyright 2019 Google Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import groovy.xml.MarkupBuilder
plugins {
id 'com.android.application'
}
def twaManifest = [
applicationId: 'dev.khoj.app',
hostName: 'app.khoj.dev', // The domain being opened in the TWA.
launchUrl: '/', // The start path for the TWA. Must be relative to the domain.
name: 'Khoj AI', // The application name.
launcherName: 'Khoj', // The name shown on the Android Launcher.
themeColor: '#FFFFFF', // The color used for the status bar.
themeColorDark: '#000000', // The color used for the dark status bar.
navigationColor: '#000000', // The color used for the navigation bar.
navigationColorDark: '#000000', // The color used for the dark navbar.
navigationDividerColor: '#000000', // The navbar divider color.
navigationDividerColorDark: '#000000', // The dark navbar divider color.
backgroundColor: '#FFFFFF', // The color used for the splash screen background.
enableNotifications: true, // Set to true to enable notification delegation.
// Every shortcut must include the following fields:
// - name: String that will show up in the shortcut.
// - short_name: Shorter string used if |name| is too long.
// - url: Absolute path of the URL to launch the app with (e.g '/create').
// - icon: Name of the resource in the drawable folder to use as an icon.
shortcuts: [],
// The duration of fade out animation in milliseconds to be played when removing splash screen.
splashScreenFadeOutDuration: 300,
generatorApp: 'bubblewrap-cli', // Application that generated the Android Project
// The fallback strategy for when Trusted Web Activity is not available. Possible values are
// 'customtabs' and 'webview'.
fallbackType: 'customtabs',
enableSiteSettingsShortcut: 'true',
orientation: 'natural',
]
android {
compileSdkVersion 35
namespace "dev.khoj.app"
defaultConfig {
applicationId "dev.khoj.app"
minSdkVersion 19
targetSdkVersion 35
versionCode 4
versionName "4"
// The name for the application
resValue "string", "appName", twaManifest.name
// The name for the application on the Android Launcher
resValue "string", "launcherName", twaManifest.launcherName
// The URL that will be used when launching the TWA from the Android Launcher
def launchUrl = "https://" + twaManifest.hostName + twaManifest.launchUrl
resValue "string", "launchUrl", launchUrl
// The URL the Web Manifest for the Progressive Web App that the TWA points to. This
// is used by Chrome OS and Meta Quest to open the Web version of the PWA instead of
// the TWA, as it will probably give a better user experience for non-mobile devices.
resValue "string", "webManifestUrl", 'https://app.khoj.dev/static/khoj.webmanifest'
// This is used by Meta Quest.
resValue "string", "fullScopeUrl", 'https://app.khoj.dev/'
// The hostname is used when building the intent-filter, so the TWA is able to
// handle Intents to open host url of the application.
resValue "string", "hostName", twaManifest.hostName
// This attribute sets the status bar color for the TWA. It can be either set here or in
// `res/values/colors.xml`. Setting in both places is an error and the app will not
// compile. If not set, the status bar color defaults to #FFFFFF - white.
resValue "color", "colorPrimary", twaManifest.themeColor
// This attribute sets the dark status bar color for the TWA. It can be either set here or in
// `res/values/colors.xml`. Setting in both places is an error and the app will not
// compile. If not set, the status bar color defaults to #000000 - white.
resValue "color", "colorPrimaryDark", twaManifest.themeColorDark
// This attribute sets the navigation bar color for the TWA. It can be either set here or
// in `res/values/colors.xml`. Setting in both places is an error and the app will not
// compile. If not set, the navigation bar color defaults to #FFFFFF - white.
resValue "color", "navigationColor", twaManifest.navigationColor
// This attribute sets the dark navigation bar color for the TWA. It can be either set here
// or in `res/values/colors.xml`. Setting in both places is an error and the app will not
// compile. If not set, the navigation bar color defaults to #000000 - black.
resValue "color", "navigationColorDark", twaManifest.navigationColorDark
// This attribute sets the navbar divider color for the TWA. It can be either
// set here or in `res/values/colors.xml`. Setting in both places is an error and the app
// will not compile. If not set, the divider color defaults to #00000000 - transparent.
resValue "color", "navigationDividerColor", twaManifest.navigationDividerColor
// This attribute sets the dark navbar divider color for the TWA. It can be either
// set here or in `res/values/colors.xml`. Setting in both places is an error and the
//app will not compile. If not set, the divider color defaults to #000000 - black.
resValue "color", "navigationDividerColorDark", twaManifest.navigationDividerColorDark
// Sets the color for the background used for the splash screen when launching the
// Trusted Web Activity.
resValue "color", "backgroundColor", twaManifest.backgroundColor
// Defines a provider authority for the Splash Screen
resValue "string", "providerAuthority", twaManifest.applicationId + '.fileprovider'
// The enableNotification resource is used to enable or disable the
// TrustedWebActivityService, by changing the android:enabled and android:exported
// attributes
resValue "bool", "enableNotification", twaManifest.enableNotifications.toString()
twaManifest.shortcuts.eachWithIndex { shortcut, index ->
resValue "string", "shortcut_name_$index", "$shortcut.name"
resValue "string", "shortcut_short_name_$index", "$shortcut.short_name"
}
// The splashScreenFadeOutDuration resource is used to set the duration of fade out animation in milliseconds
// to be played when removing splash screen. The default is 0 (no animation).
resValue "integer", "splashScreenFadeOutDuration", twaManifest.splashScreenFadeOutDuration.toString()
resValue "string", "generatorApp", twaManifest.generatorApp
resValue "string", "fallbackType", twaManifest.fallbackType
resValue "bool", "enableSiteSettingsShortcut", twaManifest.enableSiteSettingsShortcut
resValue "string", "orientation", twaManifest.orientation
}
buildTypes {
release {
minifyEnabled true
}
}
compileOptions {
sourceCompatibility JavaVersion.VERSION_1_8
targetCompatibility JavaVersion.VERSION_1_8
}
lintOptions {
checkReleaseBuilds false
}
}
task generateShorcutsFile {
assert twaManifest.shortcuts.size() < 5, "You can have at most 4 shortcuts."
twaManifest.shortcuts.eachWithIndex { s, i ->
assert s.name != null, 'Missing `name` in shortcut #' + i
assert s.short_name != null, 'Missing `short_name` in shortcut #' + i
assert s.url != null, 'Missing `icon` in shortcut #' + i
assert s.icon != null, 'Missing `url` in shortcut #' + i
}
def shortcutsFile = new File("$projectDir/src/main/res/xml", "shortcuts.xml")
def xmlWriter = new StringWriter()
def xmlMarkup = new MarkupBuilder(new IndentPrinter(xmlWriter, " ", true))
xmlMarkup
.'shortcuts'('xmlns:android': 'http://schemas.android.com/apk/res/android') {
twaManifest.shortcuts.eachWithIndex { s, i ->
'shortcut'(
'android:shortcutId': 'shortcut' + i,
'android:enabled': 'true',
'android:icon': '@drawable/' + s.icon,
'android:shortcutShortLabel': '@string/shortcut_short_name_' + i,
'android:shortcutLongLabel': '@string/shortcut_name_' + i) {
'intent'(
'android:action': 'android.intent.action.MAIN',
'android:targetPackage': twaManifest.applicationId,
'android:targetClass': twaManifest.applicationId + '.LauncherActivity',
'android:data': s.url)
'categories'('android:name': 'android.intent.category.LAUNCHER')
}
}
}
shortcutsFile.text = xmlWriter.toString() + '\n'
}
preBuild.dependsOn(generateShorcutsFile)
repositories {
}
dependencies {
implementation fileTree(include: ['*.jar'], dir: 'libs')
implementation 'com.google.androidbrowserhelper:locationdelegation:1.1.1'
implementation 'com.google.androidbrowserhelper:androidbrowserhelper:2.5.0'
}

188
src/interface/android/app/src/main/AndroidManifest.xml Normal file
View File

@@ -0,0 +1,188 @@
<!--
Copyright 2019 Google Inc. All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<!-- The "package" attribute is rewritten by the Gradle build with the value of applicationId.
It is still required here, as it is used to derive paths, for instance when referring
to an Activity by ".MyActivity" instead of the full name. If more Activities are added to the
application, the package attribute will need to reflect the correct path in order to use
the abbreviated format. -->
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
package="dev.khoj.app">
<uses-permission android:name="android.permission.POST_NOTIFICATIONS"/>
<application
android:name="Application"
android:allowBackup="true"
android:icon="@mipmap/ic_launcher"
android:label="@string/appName"
android:manageSpaceActivity="com.google.androidbrowserhelper.trusted.ManageDataLauncherActivity"
android:supportsRtl="true"
android:theme="@android:style/Theme.Translucent.NoTitleBar">
<meta-data
android:name="asset_statements"
android:resource="@string/assetStatements" />
<meta-data
android:name="web_manifest_url"
android:value="@string/webManifestUrl" />
<meta-data
android:name="twa_generator"
android:value="@string/generatorApp" />
<activity android:name="com.google.androidbrowserhelper.trusted.ManageDataLauncherActivity">
<meta-data
android:name="android.support.customtabs.trusted.MANAGE_SPACE_URL"
android:value="@string/launchUrl" />
</activity>
<activity android:name="LauncherActivity"
android:alwaysRetainTaskState="true"
android:label="@string/launcherName"
android:exported="true">
<meta-data android:name="android.support.customtabs.trusted.DEFAULT_URL"
android:value="@string/launchUrl" />
<meta-data
android:name="android.support.customtabs.trusted.STATUS_BAR_COLOR"
android:resource="@color/colorPrimary" />
<meta-data
android:name="android.support.customtabs.trusted.STATUS_BAR_COLOR_DARK"
android:resource="@color/colorPrimaryDark" />
<meta-data
android:name="android.support.customtabs.trusted.NAVIGATION_BAR_COLOR"
android:resource="@color/navigationColor" />
<meta-data
android:name="android.support.customtabs.trusted.NAVIGATION_BAR_COLOR_DARK"
android:resource="@color/navigationColorDark" />
<meta-data
android:name="androix.browser.trusted.NAVIGATION_BAR_DIVIDER_COLOR"
android:resource="@color/navigationDividerColor" />
<meta-data
android:name="androix.browser.trusted.NAVIGATION_BAR_DIVIDER_COLOR_DARK"
android:resource="@color/navigationDividerColorDark" />
<meta-data android:name="android.support.customtabs.trusted.SPLASH_IMAGE_DRAWABLE"
android:resource="@drawable/splash"/>
<meta-data android:name="android.support.customtabs.trusted.SPLASH_SCREEN_BACKGROUND_COLOR"
android:resource="@color/backgroundColor"/>
<meta-data android:name="android.support.customtabs.trusted.SPLASH_SCREEN_FADE_OUT_DURATION"
android:value="@integer/splashScreenFadeOutDuration"/>
<meta-data android:name="android.support.customtabs.trusted.FILE_PROVIDER_AUTHORITY"
android:value="@string/providerAuthority"/>
<meta-data android:name="android.app.shortcuts" android:resource="@xml/shortcuts" />
<meta-data android:name="android.support.customtabs.trusted.FALLBACK_STRATEGY"
android:value="@string/fallbackType" />
<meta-data android:name="android.support.customtabs.trusted.SCREEN_ORIENTATION"
android:value="@string/orientation"/>
<intent-filter>
<action android:name="android.intent.action.MAIN" />
<category android:name="android.intent.category.LAUNCHER" />
</intent-filter>
<intent-filter android:autoVerify="true">
<action android:name="android.intent.action.VIEW"/>
<category android:name="android.intent.category.DEFAULT" />
<category android:name="android.intent.category.BROWSABLE"/>
<data android:scheme="https"
android:host="@string/hostName"/>
</intent-filter>
</activity>
<activity android:name="com.google.androidbrowserhelper.trusted.FocusActivity" />
<activity android:name="com.google.androidbrowserhelper.trusted.WebViewFallbackActivity"
android:configChanges="orientation|screenSize" />
<provider
android:name="androidx.core.content.FileProvider"
android:authorities="@string/providerAuthority"
android:grantUriPermissions="true"
android:exported="false">
<meta-data
android:name="android.support.FILE_PROVIDER_PATHS"
android:resource="@xml/filepaths" />
</provider>
<service
android:name=".DelegationService"
android:enabled="@bool/enableNotification"
android:exported="@bool/enableNotification">
<meta-data
android:name="android.support.customtabs.trusted.SMALL_ICON"
android:resource="@drawable/ic_notification_icon" />
<intent-filter>
<action android:name="android.support.customtabs.trusted.TRUSTED_WEB_ACTIVITY_SERVICE"/>
<category android:name="android.intent.category.DEFAULT"/>
</intent-filter>
</service>
<activity android:name="com.google.androidbrowserhelper.trusted.NotificationPermissionRequestActivity" />
<activity android:name=
"com.google.androidbrowserhelper.locationdelegation.PermissionRequestActivity"/>
</application>
</manifest>

29
src/interface/android/app/src/main/java/dev/khoj/app/Application.java Normal file
View File

@@ -0,0 +1,29 @@
/*
* Copyright 2020 Google Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package dev.khoj.app;
public class Application extends android.app.Application {
@Override
public void onCreate() {
super.onCreate();
}
}

17
src/interface/android/app/src/main/java/dev/khoj/app/DelegationService.java Normal file
View File

@@ -0,0 +1,17 @@
package dev.khoj.app;
import com.google.androidbrowserhelper.locationdelegation.LocationDelegationExtraCommandHandler;
public class DelegationService extends
com.google.androidbrowserhelper.trusted.DelegationService {
@Override
public void onCreate() {
super.onCreate();
registerExtraCommandHandler(new LocationDelegationExtraCommandHandler());
}
}

54
src/interface/android/app/src/main/java/dev/khoj/app/LauncherActivity.java Normal file
View File

@@ -0,0 +1,54 @@
/*
* Copyright 2020 Google Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package dev.khoj.app;
import android.content.pm.ActivityInfo;
import android.net.Uri;
import android.os.Build;
import android.os.Bundle;
public class LauncherActivity
extends com.google.androidbrowserhelper.trusted.LauncherActivity {
@Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
// Setting an orientation crashes the app due to the transparent background on Android 8.0
// Oreo and below. We only set the orientation on Oreo and above. This only affects the
// splash screen and Chrome will still respect the orientation.
// See https://github.com/GoogleChromeLabs/bubblewrap/issues/496 for details.
if (Build.VERSION.SDK_INT > Build.VERSION_CODES.O) {
setRequestedOrientation(ActivityInfo.SCREEN_ORIENTATION_UNSPECIFIED);
} else {
setRequestedOrientation(ActivityInfo.SCREEN_ORIENTATION_UNSPECIFIED);
}
}
@Override
protected Uri getLaunchingUrl() {
// Get the original launch Url.
Uri uri = super.getLaunchingUrl();
return uri;
}
}

25
src/interface/android/app/src/main/res/drawable-anydpi/shortcut_legacy_background.xml Normal file
View File

@@ -0,0 +1,25 @@
<!--
Copyright 2020 Google Inc. All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<inset xmlns:android="http://schemas.android.com/apk/res/android"
xmlns:aapt="http://schemas.android.com/aapt"
android:inset="2dp">
<aapt:attr name="android:drawable">
<shape android:shape="oval">
<solid android:color="@color/shortcut_background" />
<size android:width="44dp" android:height="44dp" />
</shape>
</aapt:attr>
</inset>

BIN
src/interface/android/app/src/main/res/drawable-hdpi/ic_notification_icon.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.1 KiB

BIN
src/interface/android/app/src/main/res/drawable-hdpi/splash.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 21 KiB

BIN
src/interface/android/app/src/main/res/drawable-mdpi/ic_notification_icon.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 678 B

BIN
src/interface/android/app/src/main/res/drawable-mdpi/splash.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 13 KiB

BIN
src/interface/android/app/src/main/res/drawable-xhdpi/ic_notification_icon.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.5 KiB

BIN
src/interface/android/app/src/main/res/drawable-xhdpi/splash.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 26 KiB

BIN
src/interface/android/app/src/main/res/drawable-xxhdpi/ic_notification_icon.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 2.4 KiB

BIN
src/interface/android/app/src/main/res/drawable-xxhdpi/splash.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 50 KiB

BIN
src/interface/android/app/src/main/res/drawable-xxxhdpi/ic_notification_icon.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 3.4 KiB

BIN
src/interface/android/app/src/main/res/drawable-xxxhdpi/splash.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 60 KiB

BIN
src/interface/android/app/src/main/res/mipmap-hdpi/ic_launcher.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 3.0 KiB

BIN
src/interface/android/app/src/main/res/mipmap-mdpi/ic_launcher.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.7 KiB

BIN
src/interface/android/app/src/main/res/mipmap-xhdpi/ic_launcher.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 4.3 KiB

BIN
src/interface/android/app/src/main/res/mipmap-xxhdpi/ic_launcher.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 8.0 KiB

BIN
src/interface/android/app/src/main/res/mipmap-xxxhdpi/ic_launcher.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 12 KiB

1
src/interface/android/app/src/main/res/raw/web_app_manifest.json Normal file
View File

@@ -0,0 +1 @@
{"name":"Khoj","short_name":"Khoj","display":"standalone","start_url":"/","description":"The open, personal AI for your digital brain. You can ask Khoj to draft a message, paint your imagination, find information on the internet and even answer questions from your documents.","theme_color":"#ffffff","background_color":"#ffffff","icons":[{"src":"/static/assets/icons/khoj_lantern_128x128.png","sizes":"128x128","type":"image/png"},{"src":"/static/assets/icons/khoj_lantern_256x256.png","sizes":"256x256","type":"image/png"}],"screenshots":[{"src":"/static/assets/samples/phone-remember-plan-sample.png","sizes":"419x900","type":"image/png","form_factor":"narrow","label":"Remember and Plan"},{"src":"/static/assets/samples/phone-browse-draw-sample.png","sizes":"419x900","type":"image/png","form_factor":"narrow","label":"Browse and Draw"},{"src":"/static/assets/samples/desktop-remember-plan-sample.png","sizes":"1260x742","type":"image/png","form_factor":"wide","label":"Remember and Plan"},{"src":"/static/assets/samples/desktop-browse-draw-sample.png","sizes":"1260x742","type":"image/png","form_factor":"wide","label":"Browse and Draw"}]}

18
src/interface/android/app/src/main/res/values/colors.xml Normal file
View File

@@ -0,0 +1,18 @@
<!--
Copyright 2020 Google Inc. All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<resources>
<color name="shortcut_background">#F5F5F5</color>
</resources>

36
src/interface/android/app/src/main/res/values/strings.xml Normal file
View File

@@ -0,0 +1,36 @@
<?xml version="1.0" encoding="utf-8"?>
<!--
Copyright 2021 Google Inc. All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<resources>
<!--
This variable below expresses the relationship between the app and the site,
as documented in the TWA documentation at
https://developers.google.com/web/updates/2017/10/using-twa#set_up_digital_asset_links_in_an_android_app
and is injected into the AndroidManifest.xml
-->
<string name="assetStatements">
[{
\"relation\": [\"delegate_permission/common.handle_all_urls\"],
\"target\": {
\"namespace\": \"web\",
\"site\": \"https://app.khoj.dev\"
}
}]
</string>
</resources>

18
src/interface/android/app/src/main/res/xml/filepaths.xml Normal file
View File

@@ -0,0 +1,18 @@
<!--
Copyright 2019 Google Inc. All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<paths>
<files-path path="twa_splash/" name="twa_splash" />
</paths>

1
src/interface/android/app/src/main/res/xml/shortcuts.xml Normal file
View File

@@ -0,0 +1 @@
<shortcuts xmlns:android='http://schemas.android.com/apk/res/android' />

42
src/interface/android/build.gradle Normal file
View File

@@ -0,0 +1,42 @@
/*
* Copyright 2019 Google Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
// Top-level build file where you can add configuration options common to all sub-projects/modules.
buildscript {
repositories {
google()
mavenCentral()
}
dependencies {
classpath 'com.android.tools.build:gradle:8.7.2'
// NOTE: Do not place your application dependencies here; they belong
// in the individual module build.gradle files
}
}
allprojects {
repositories {
google()
mavenCentral()
}
}
tasks.register('clean', Delete) {
delete rootProject.buildDir
}

14
src/interface/android/gradle.properties Normal file
View File

@@ -0,0 +1,14 @@
# Project-wide Gradle settings.
# IDE (e.g. Android Studio) users:
# Gradle settings configured through the IDE *will override*
# any settings specified in this file.
# For more details on how to configure your build environment visit
# http://www.gradle.org/docs/current/userguide/build_environment.html
# Specifies the JVM arguments used for the daemon process.
# The setting is particularly useful for tweaking memory settings.
org.gradle.jvmargs=-Xmx1536m
# When configured, Gradle will run in incubating parallel mode.
# This option should only be used with decoupled projects. More details, visit
# http://www.gradle.org/docs/current/userguide/multi_project_builds.html#sec:decoupled_projects
# org.gradle.parallel=true
android.useAndroidX=true

BIN
src/interface/android/gradle/wrapper/gradle-wrapper.jar vendored Normal file
View File

Binary file not shown.

7
src/interface/android/gradle/wrapper/gradle-wrapper.properties vendored Normal file
View File

@@ -0,0 +1,7 @@
distributionBase=GRADLE_USER_HOME
distributionPath=wrapper/dists
zipStoreBase=GRADLE_USER_HOME
zipStorePath=wrapper/dists
distributionUrl=https\://services.gradle.org/distributions/gradle-8.9-bin.zip
networkTimeout=10000
validateDistributionUrl=true

249
src/interface/android/gradlew vendored Executable file
View File

@@ -0,0 +1,249 @@
#!/bin/sh
#
# Copyright © 2015-2021 the original authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
##############################################################################
#
# Gradle start up script for POSIX generated by Gradle.
#
# Important for running:
#
# (1) You need a POSIX-compliant shell to run this script. If your /bin/sh is
# noncompliant, but you have some other compliant shell such as ksh or
# bash, then to run this script, type that shell name before the whole
# command line, like:
#
# ksh Gradle
#
# Busybox and similar reduced shells will NOT work, because this script
# requires all of these POSIX shell features:
# * functions;
# * expansions «$var», «${var}», «${var:-default}», «${var+SET}»,
# «${var#prefix}», «${var%suffix}», and «$( cmd )»;
# * compound commands having a testable exit status, especially «case»;
# * various built-in commands including «command», «set», and «ulimit».
#
# Important for patching:
#
# (2) This script targets any POSIX shell, so it avoids extensions provided
# by Bash, Ksh, etc; in particular arrays are avoided.
#
# The "traditional" practice of packing multiple parameters into a
# space-separated string is a well documented source of bugs and security
# problems, so this is (mostly) avoided, by progressively accumulating
# options in "$@", and eventually passing that to Java.
#
# Where the inherited environment variables (DEFAULT_JVM_OPTS, JAVA_OPTS,
# and GRADLE_OPTS) rely on word-splitting, this is performed explicitly;
# see the in-line comments for details.
#
# There are tweaks for specific operating systems such as AIX, CygWin,
# Darwin, MinGW, and NonStop.
#
# (3) This script is generated from the Groovy template
# https://github.com/gradle/gradle/blob/HEAD/platforms/jvm/plugins-application/src/main/resources/org/gradle/api/internal/plugins/unixStartScript.txt
# within the Gradle project.
#
# You can find Gradle at https://github.com/gradle/gradle/.
#
##############################################################################
# Attempt to set APP_HOME
# Resolve links: $0 may be a link
app_path=$0
# Need this for daisy-chained symlinks.
while
APP_HOME=${app_path%"${app_path##*/}"} # leaves a trailing /; empty if no leading path
[ -h "$app_path" ]
do
ls=$( ls -ld "$app_path" )
link=${ls#*' -> '}
case $link in #(
/*) app_path=$link ;; #(
*) app_path=$APP_HOME$link ;;
esac
done
# This is normally unused
# shellcheck disable=SC2034
APP_BASE_NAME=${0##*/}
# Discard cd standard output in case $CDPATH is set (https://github.com/gradle/gradle/issues/25036)
APP_HOME=$( cd "${APP_HOME:-./}" > /dev/null && pwd -P ) || exit
# Use the maximum available, or set MAX_FD != -1 to use that value.
MAX_FD=maximum
warn () {
echo "$*"
} >&2
die () {
echo
echo "$*"
echo
exit 1
} >&2
# OS specific support (must be 'true' or 'false').
cygwin=false
msys=false
darwin=false
nonstop=false
case "$( uname )" in #(
CYGWIN* ) cygwin=true ;; #(
Darwin* ) darwin=true ;; #(
MSYS* | MINGW* ) msys=true ;; #(
NONSTOP* ) nonstop=true ;;
esac
CLASSPATH=$APP_HOME/gradle/wrapper/gradle-wrapper.jar
# Determine the Java command to use to start the JVM.
if [ -n "$JAVA_HOME" ] ; then
if [ -x "$JAVA_HOME/jre/sh/java" ] ; then
# IBM's JDK on AIX uses strange locations for the executables
JAVACMD=$JAVA_HOME/jre/sh/java
else
JAVACMD=$JAVA_HOME/bin/java
fi
if [ ! -x "$JAVACMD" ] ; then
die "ERROR: JAVA_HOME is set to an invalid directory: $JAVA_HOME
Please set the JAVA_HOME variable in your environment to match the
location of your Java installation."
fi
else
JAVACMD=java
if ! command -v java >/dev/null 2>&1
then
die "ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH.
Please set the JAVA_HOME variable in your environment to match the
location of your Java installation."
fi
fi
# Increase the maximum file descriptors if we can.
if ! "$cygwin" && ! "$darwin" && ! "$nonstop" ; then
case $MAX_FD in #(
max*)
# In POSIX sh, ulimit -H is undefined. That's why the result is checked to see if it worked.
# shellcheck disable=SC2039,SC3045
MAX_FD=$( ulimit -H -n ) ||
warn "Could not query maximum file descriptor limit"
esac
case $MAX_FD in #(
'' | soft) :;; #(
*)
# In POSIX sh, ulimit -n is undefined. That's why the result is checked to see if it worked.
# shellcheck disable=SC2039,SC3045
ulimit -n "$MAX_FD" ||
warn "Could not set maximum file descriptor limit to $MAX_FD"
esac
fi
# Collect all arguments for the java command, stacking in reverse order:
# * args from the command line
# * the main class name
# * -classpath
# * -D...appname settings
# * --module-path (only if needed)
# * DEFAULT_JVM_OPTS, JAVA_OPTS, and GRADLE_OPTS environment variables.
# For Cygwin or MSYS, switch paths to Windows format before running java
if "$cygwin" || "$msys" ; then
APP_HOME=$( cygpath --path --mixed "$APP_HOME" )
CLASSPATH=$( cygpath --path --mixed "$CLASSPATH" )
JAVACMD=$( cygpath --unix "$JAVACMD" )
# Now convert the arguments - kludge to limit ourselves to /bin/sh
for arg do
if
case $arg in #(
-*) false ;; # don't mess with options #(
/?*) t=${arg#/} t=/${t%%/*} # looks like a POSIX filepath
[ -e "$t" ] ;; #(
*) false ;;
esac
then
arg=$( cygpath --path --ignore --mixed "$arg" )
fi
# Roll the args list around exactly as many times as the number of
# args, so each arg winds up back in the position where it started, but
# possibly modified.
#
# NB: a `for` loop captures its iteration list before it begins, so
# changing the positional parameters here affects neither the number of
# iterations, nor the values presented in `arg`.
shift # remove old arg
set -- "$@" "$arg" # push replacement arg
done
fi
# Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script.
DEFAULT_JVM_OPTS='"-Xmx64m" "-Xms64m"'
# Collect all arguments for the java command:
# * DEFAULT_JVM_OPTS, JAVA_OPTS, JAVA_OPTS, and optsEnvironmentVar are not allowed to contain shell fragments,
# and any embedded shellness will be escaped.
# * For example: A user cannot expect ${Hostname} to be expanded, as it is an environment variable and will be
# treated as '${Hostname}' itself on the command line.
set -- \
"-Dorg.gradle.appname=$APP_BASE_NAME" \
-classpath "$CLASSPATH" \
org.gradle.wrapper.GradleWrapperMain \
"$@"
# Stop when "xargs" is not available.
if ! command -v xargs >/dev/null 2>&1
then
die "xargs is not available"
fi
# Use "xargs" to parse quoted args.
#
# With -n1 it outputs one arg per line, with the quotes and backslashes removed.
#
# In Bash we could simply go:
#
# readarray ARGS < <( xargs -n1 <<<"$var" ) &&
# set -- "${ARGS[@]}" "$@"
#
# but POSIX shell has neither arrays nor command substitution, so instead we
# post-process each arg (as a line of input to sed) to backslash-escape any
# character that might be a shell metacharacter, then use eval to reverse
# that process (while maintaining the separation between arguments), and wrap
# the whole thing up as a single "set" statement.
#
# This will of course break if any of these variables contains a newline or
# an unmatched quote.
#
eval "set -- $(
printf '%s\n' "$DEFAULT_JVM_OPTS $JAVA_OPTS $GRADLE_OPTS" |
xargs -n1 |
sed ' s~[^-[:alnum:]+,./:=@_]~\\&~g; ' |
tr '\n' ' '
)" '"$@"'
exec "$JAVACMD" "$@"

92
src/interface/android/gradlew.bat vendored Normal file
View File

@@ -0,0 +1,92 @@
@rem
@rem Copyright 2015 the original author or authors.
@rem
@rem Licensed under the Apache License, Version 2.0 (the "License");
@rem you may not use this file except in compliance with the License.
@rem You may obtain a copy of the License at
@rem
@rem https://www.apache.org/licenses/LICENSE-2.0
@rem
@rem Unless required by applicable law or agreed to in writing, software
@rem distributed under the License is distributed on an "AS IS" BASIS,
@rem WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
@rem See the License for the specific language governing permissions and
@rem limitations under the License.
@rem
@if "%DEBUG%"=="" @echo off
@rem ##########################################################################
@rem
@rem Gradle startup script for Windows
@rem
@rem ##########################################################################
@rem Set local scope for the variables with windows NT shell
if "%OS%"=="Windows_NT" setlocal
set DIRNAME=%~dp0
if "%DIRNAME%"=="" set DIRNAME=.
@rem This is normally unused
set APP_BASE_NAME=%~n0
set APP_HOME=%DIRNAME%
@rem Resolve any "." and ".." in APP_HOME to make it shorter.
for %%i in ("%APP_HOME%") do set APP_HOME=%%~fi
@rem Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script.
set DEFAULT_JVM_OPTS="-Xmx64m" "-Xms64m"
@rem Find java.exe
if defined JAVA_HOME goto findJavaFromJavaHome
set JAVA_EXE=java.exe
%JAVA_EXE% -version >NUL 2>&1
if %ERRORLEVEL% equ 0 goto execute
echo. 1>&2
echo ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH. 1>&2
echo. 1>&2
echo Please set the JAVA_HOME variable in your environment to match the 1>&2
echo location of your Java installation. 1>&2
goto fail
:findJavaFromJavaHome
set JAVA_HOME=%JAVA_HOME:"=%
set JAVA_EXE=%JAVA_HOME%/bin/java.exe
if exist "%JAVA_EXE%" goto execute
echo. 1>&2
echo ERROR: JAVA_HOME is set to an invalid directory: %JAVA_HOME% 1>&2
echo. 1>&2
echo Please set the JAVA_HOME variable in your environment to match the 1>&2
echo location of your Java installation. 1>&2
goto fail
:execute
@rem Setup the command line
set CLASSPATH=%APP_HOME%\gradle\wrapper\gradle-wrapper.jar
@rem Execute Gradle
"%JAVA_EXE%" %DEFAULT_JVM_OPTS% %JAVA_OPTS% %GRADLE_OPTS% "-Dorg.gradle.appname=%APP_BASE_NAME%" -classpath "%CLASSPATH%" org.gradle.wrapper.GradleWrapperMain %*
:end
@rem End local scope for the variables with windows NT shell
if %ERRORLEVEL% equ 0 goto mainEnd
:fail
rem Set variable GRADLE_EXIT_CONSOLE if you need the _script_ return code instead of
rem the _cmd.exe /c_ return code!
set EXIT_CODE=%ERRORLEVEL%
if %EXIT_CODE% equ 0 set EXIT_CODE=1
if not ""=="%GRADLE_EXIT_CONSOLE%" exit %EXIT_CODE%
exit /b %EXIT_CODE%
:mainEnd
if "%OS%"=="Windows_NT" endlocal
:omega

1
src/interface/android/manifest-checksum.txt Normal file
View File

@@ -0,0 +1 @@
6ee1711cf4f745dafc80c1cc13c3025342a0f5da

1
src/interface/android/settings.gradle Normal file
View File

@@ -0,0 +1 @@
include ':app'

BIN
src/interface/android/store_icon.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 25 KiB

55
src/interface/android/twa-manifest.json Normal file
View File

@@ -0,0 +1,55 @@
{
"packageId": "dev.khoj.app",
"host": "app.khoj.dev",
"name": "Khoj AI",
"launcherName": "Khoj",
"display": "standalone",
"themeColor": "#FFFFFF",
"themeColorDark": "#000000",
"navigationColor": "#000000",
"navigationColorDark": "#000000",
"navigationDividerColor": "#000000",
"navigationDividerColorDark": "#000000",
"backgroundColor": "#FFFFFF",
"enableNotifications": true,
"startUrl": "/",
"iconUrl": "https://assets.khoj.dev/khoj_lantern_1200x1200.png",
"splashScreenFadeOutDuration": 300,
"signingKey": {
"path": "android.keystore",
"alias": "android"
},
"appVersionName": "4",
"appVersionCode": 4,
"shortcuts": [],
"generatorApp": "bubblewrap-cli",
"webManifestUrl": "https://app.khoj.dev/static/khoj.webmanifest",
"fallbackType": "customtabs",
"features": {
"locationDelegation": {
"enabled": true
}
},
"alphaDependencies": {
"enabled": false
},
"enableSiteSettingsShortcut": true,
"isChromeOSOnly": false,
"isMetaQuest": false,
"fullScopeUrl": "https://app.khoj.dev/",
"minSdkVersion": 19,
"orientation": "natural",
"fingerprints": [
{
"name": "signing",
"value": "CC:98:4A:0A:F1:CC:84:26:AC:02:86:49:AA:69:64:B9:5E:63:A3:EF:18:56:EA:CA:13:C1:3A:15:CA:49:77:46"
},
{
"name": "upload",
"value": "D4:5A:6F:6C:18:28:D2:1C:78:27:92:C6:AC:DB:4C:12:C4:52:A1:88:9B:A1:F5:67:D1:22:FE:A0:0F:B1:AE:92"
}
],
"additionalTrustedOrigins": [],
"retainedBundles": [],
"appVersion": "4"
}

24
src/interface/desktop/assets/icons/chat.svg
View File

@@ -1,24 +0,0 @@
<svg xmlns="http://www.w3.org/2000/svg"
width="800px"
height="800px"
viewBox="0 0 24 24"
fill="none"
version="1.1">
<path
d="m 14.024348,9.8497703 0.04627,1.9750167"
stroke="#1c274c"
stroke-width="1.77073"
stroke-linecap="round" />
<path
d="m 9.6453624,9.7953624 0.046275,1.9750166"
stroke="#1c274c"
stroke-width="1.77072"
stroke-linecap="round" />
<path
d="m 11.90538,2.3619994 c -5.4939109,0 -9.6890976,4.0608185 -9.6890976,9.8578926 0,1.477202 0.2658016,2.542848 0.6989332,3.331408 0.433559,0.789293 1.0740097,1.372483 1.9230615,1.798517 1.7362861,0.87132 4.1946007,1.018626 7.0671029,1.018626 0.317997,0 0.593711,0.167879 0.784844,0.458501 0.166463,0.253124 0.238617,0.552748 0.275566,0.787233 0.07263,0.460801 0.05871,1.030165 0.04785,1.474824 v 4.8e-5 l -2.26e-4,0.0091 c -0.0085,0.348246 -0.01538,0.634247 -0.0085,0.861186 0.105589,-0.07971 0.227925,-0.185287 0.36735,-0.31735 0.348613,-0.330307 0.743513,-0.767362 1.176607,-1.246635 l 0.07837,-0.08673 c 0.452675,-0.500762 0.941688,-1.037938 1.41216,-1.473209 0.453774,-0.419787 0.969948,-0.822472 1.476003,-0.953853 1.323661,-0.343655 2.330132,-0.904027 3.005749,-1.76381 0.658957,-0.838568 1.073167,-2.051868 1.073167,-3.898667 0,-5.7970748 -4.195186,-9.8578946 -9.689097,-9.8578946 z M 0.92440678,12.219892 c 0,-7.0067939 5.05909412,-11.47090892 10.98097322,-11.47090892 5.921878,0 10.980972,4.46411502 10.980972,11.47090892 0,2.172259 -0.497596,3.825405 -1.442862,5.028357 -0.928601,1.181693 -2.218843,1.837914 -3.664937,2.213334 -0.211641,0.05502 -0.53529,0.268579 -0.969874,0.670658 -0.417861,0.386604 -0.865628,0.876836 -1.324566,1.384504 l -0.09131,0.101202 c -0.419252,0.464136 -0.849637,0.94059 -1.239338,1.309807 -0.210187,0.199169 -0.425281,0.383422 -0.635348,0.523424 -0.200911,0.133819 -0.449635,0.263369 -0.716376,0.281474 -0.327812,0.02226 -0.61539,-0.149209 -0.804998,-0.457293 -0.157614,-0.255993 -0.217622,-0.557143 -0.246564,-0.778198 -0.0542,-0.414027 -0.04101,-0.933065 -0.03027,-1.355183 l 0.0024,-0.0922 c 0.01099,-0.463865 0.01489,-0.820507 -0.01611,-1.06842 C 8.9434608,19.975238 6.3139711,19.828758 4.356743,18.84659 3.3355029,18.334136 2.4624526,17.578678 1.8500164,16.463713 1.2372016,15.348029 0.92459928,13.943803 0.92459928,12.219967 Z"
clip-rule="evenodd"
stroke-width="0.360886"
fill="#1c274c"
fill-rule="evenodd"
fill-opacity="1" />
</svg>
Side by Side

Before

Width:  |  Height:  |  Size: 2.4 KiB

5
src/interface/desktop/assets/icons/circular-add.svg
View File

@@ -1,5 +0,0 @@
<?xml version="1.0" encoding="utf-8"?><!-- Uploaded to: SVG Repo, www.svgrepo.com, Generator: SVG Repo Mixer Tools -->
<svg width="800px" height="800px" viewBox="0 0 24 24" fill="none" xmlns="http://www.w3.org/2000/svg">
<path d="M15 12L12 12M12 12L9 12M12 12L12 9M12 12L12 15" stroke="#1C274C" stroke-width="1.5" stroke-linecap="round"/>
<path d="M7 3.33782C8.47087 2.48697 10.1786 2 12 2C17.5228 2 22 6.47715 22 12C22 17.5228 17.5228 22 12 22C6.47715 22 2 17.5228 2 12C2 10.1786 2.48697 8.47087 3.33782 7" stroke="#1C274C" stroke-width="1.5" stroke-linecap="round"/>
</svg>
Side by Side

Before

Width:  |  Height:  |  Size: 580 B

1
src/interface/desktop/assets/icons/clock.svg Normal file
View File

@@ -0,0 +1 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 256 256"><rect width="256" height="256" fill="none"/><polyline points="128 80 128 128 168 152" fill="none" stroke="currentColor" stroke-linecap="round" stroke-linejoin="round" stroke-width="16"/><polyline points="184 104 224 104 224 64" fill="none" stroke="currentColor" stroke-linecap="round" stroke-linejoin="round" stroke-width="16"/><path d="M188.4,192a88,88,0,1,1,1.83-126.23C202,77.69,211.72,88.93,224,104" fill="none" stroke="currentColor" stroke-linecap="round" stroke-linejoin="round" stroke-width="16"/></svg>
Side by Side

After

Width:  |  Height:  |  Size: 573 B

1
src/interface/desktop/assets/icons/confirm-icon.svg
View File

@@ -1 +0,0 @@
<svg id="Layer_1" data-name="Layer 1" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 122.88 122.88"><defs><style>.cls-1{fill:#00a912;}.cls-1,.cls-2{fill-rule:evenodd;}.cls-2{fill:#fff;}</style></defs><title>confirm</title><path class="cls-1" d="M61.44,0A61.44,61.44,0,1,1,0,61.44,61.44,61.44,0,0,1,61.44,0Z"/><path class="cls-2" d="M42.37,51.68,53.26,62,79,35.87c2.13-2.16,3.47-3.9,6.1-1.19l8.53,8.74c2.8,2.77,2.66,4.4,0,7L58.14,85.34c-5.58,5.46-4.61,5.79-10.26.19L28,65.77c-1.18-1.28-1.05-2.57.24-3.84l9.9-10.27c1.5-1.58,2.7-1.44,4.22,0Z"/></svg>
Side by Side

Before

Width:  |  Height:  |  Size: 549 B

10
src/interface/desktop/assets/icons/file-not-synced.svg
View File

@@ -1,9 +1 @@
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<!-- Uploaded to: SVG Repo, www.svgrepo.com, Transformed by: SVG Repo Mixer Tools -->
<svg width="800px" height="800px" viewBox="-2.4 -2.4 28.80 28.80" fill="none" xmlns="http://www.w3.org/2000/svg">
<g id="SVGRepo_bgCarrier" stroke-width="0">
<rect x="-2.4" y="-2.4" width="28.80" height="28.80" rx="14.4" fill="#ffad9f" strokewidth="0"/>
</g>
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 256 256"><rect width="256" height="256" fill="none"/><line x1="48" y1="40" x2="208" y2="216" fill="none" stroke="currentColor" stroke-linecap="round" stroke-linejoin="round" stroke-width="16"/><path d="M193.89,200.49A79.66,79.66,0,0,1,160,208H72A56,56,0,1,1,85.92,97.74" fill="none" stroke="currentColor" stroke-linecap="round" stroke-linejoin="round" stroke-width="16"/><path d="M112.63,63.52A80,80,0,0,1,219.68,181.28" fill="none" stroke="currentColor" stroke-linecap="round" stroke-linejoin="round" stroke-width="16"/><path d="M80,128A79.68,79.68,0,0,1,91.07,87.37" fill="none" stroke="currentColor" stroke-linecap="round" stroke-linejoin="round" stroke-width="16"/></svg>
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 1.6 KiB

After

Width:  |  Height:  |  Size: 729 B

10
src/interface/desktop/assets/icons/file-synced.svg
View File

@@ -1,9 +1 @@
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<!-- Uploaded to: SVG Repo, www.svgrepo.com, Transformed by: SVG Repo Mixer Tools -->
<svg width="800px" height="800px" viewBox="-2.4 -2.4 28.80 28.80" fill="none" xmlns="http://www.w3.org/2000/svg" stroke="#16ba00">
<g id="SVGRepo_bgCarrier" stroke-width="0">
<rect x="-2.4" y="-2.4" width="28.80" height="28.80" rx="14.4" fill="#7aff00" strokewidth="0"/>
</g>
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 256 256"><rect width="256" height="256" fill="none"/><path d="M80,128a80,80,0,1,1,80,80H72A56,56,0,1,1,85.92,97.74" fill="none" stroke="currentColor" stroke-linecap="round" stroke-linejoin="round" stroke-width="16"/><polyline points="120 136 144 160 192 112" fill="none" stroke="currentColor" stroke-linecap="round" stroke-linejoin="round" stroke-width="16"/></svg>
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 1.3 KiB

After

Width:  |  Height:  |  Size: 419 B

28
src/interface/desktop/assets/icons/folder.svg
View File

@@ -1,28 +0,0 @@
<?xml version="1.0" encoding="iso-8859-1"?>
<!-- Uploaded to: SVG Repo, www.svgrepo.com, Generator: SVG Repo Mixer Tools -->
<svg version="1.1" id="Layer_1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"
viewBox="0 0 512 512" xml:space="preserve">
<path id="SVGCleanerId_0" style="fill:#FFC36E;" d="M183.295,123.586H55.05c-6.687,0-12.801-3.778-15.791-9.76l-12.776-25.55
l12.776-25.55c2.99-5.982,9.103-9.76,15.791-9.76h128.246c6.687,0,12.801,3.778,15.791,9.76l12.775,25.55l-12.776,25.55
C196.096,119.808,189.983,123.586,183.295,123.586z"/>
<g>
<path id="SVGCleanerId_0_1_" style="fill:#FFC36E;" d="M183.295,123.586H55.05c-6.687,0-12.801-3.778-15.791-9.76l-12.776-25.55
l12.776-25.55c2.99-5.982,9.103-9.76,15.791-9.76h128.246c6.687,0,12.801,3.778,15.791,9.76l12.775,25.55l-12.776,25.55
C196.096,119.808,189.983,123.586,183.295,123.586z"/>
</g>
<path style="fill:#EFF2FA;" d="M485.517,70.621H26.483c-4.875,0-8.828,3.953-8.828,8.828v44.138h476.69V79.448
C494.345,74.573,490.392,70.621,485.517,70.621z"/>
<rect x="17.655" y="105.931" style="fill:#E1E6F2;" width="476.69" height="17.655"/>
<path style="fill:#FFD782;" d="M494.345,88.276H217.318c-3.343,0-6.4,1.889-7.895,4.879l-10.336,20.671
c-2.99,5.982-9.105,9.76-15.791,9.76H55.05c-6.687,0-12.801-3.778-15.791-9.76L28.922,93.155c-1.495-2.99-4.552-4.879-7.895-4.879
h-3.372C7.904,88.276,0,96.18,0,105.931v335.448c0,9.751,7.904,17.655,17.655,17.655h476.69c9.751,0,17.655-7.904,17.655-17.655
V105.931C512,96.18,504.096,88.276,494.345,88.276z"/>
<path style="fill:#FFC36E;" d="M485.517,441.379H26.483c-4.875,0-8.828-3.953-8.828-8.828l0,0c0-4.875,3.953-8.828,8.828-8.828
h459.034c4.875,0,8.828,3.953,8.828,8.828l0,0C494.345,437.427,490.392,441.379,485.517,441.379z"/>
<path style="fill:#EFF2FA;" d="M326.621,220.69h132.414c4.875,0,8.828-3.953,8.828-8.828v-70.621c0-4.875-3.953-8.828-8.828-8.828
H326.621c-4.875,0-8.828,3.953-8.828,8.828v70.621C317.793,216.737,321.746,220.69,326.621,220.69z"/>
<path style="fill:#C7CFE2;" d="M441.379,167.724h-97.103c-4.875,0-8.828-3.953-8.828-8.828l0,0c0-4.875,3.953-8.828,8.828-8.828
h97.103c4.875,0,8.828,3.953,8.828,8.828l0,0C450.207,163.772,446.254,167.724,441.379,167.724z"/>
<path style="fill:#D7DEED;" d="M441.379,203.034h-97.103c-4.875,0-8.828-3.953-8.828-8.828l0,0c0-4.875,3.953-8.828,8.828-8.828
h97.103c4.875,0,8.828,3.953,8.828,8.828l0,0C450.207,199.082,446.254,203.034,441.379,203.034z"/>
</svg>
Side by Side

Before

Width:  |  Height:  |  Size: 2.4 KiB

4
src/interface/desktop/assets/icons/key.svg
View File

@@ -1,4 +0,0 @@
<?xml version="1.0" encoding="utf-8"?>
<svg width="800px" height="800px" viewBox="0 0 24 24" fill="none" xmlns="http://www.w3.org/2000/svg">
<path fill-rule="evenodd" clip-rule="evenodd" d="M22 8.29344C22 11.7692 19.1708 14.5869 15.6807 14.5869C15.0439 14.5869 13.5939 14.4405 12.8885 13.8551L12.0067 14.7333C11.4883 15.2496 11.6283 15.4016 11.8589 15.652C11.9551 15.7565 12.0672 15.8781 12.1537 16.0505C12.1537 16.0505 12.8885 17.075 12.1537 18.0995C11.7128 18.6849 10.4783 19.5045 9.06754 18.0995L8.77362 18.3922C8.77362 18.3922 9.65538 19.4167 8.92058 20.4412C8.4797 21.0267 7.30403 21.6121 6.27531 20.5876L5.2466 21.6121C4.54119 22.3146 3.67905 21.9048 3.33616 21.6121L2.45441 20.7339C1.63143 19.9143 2.1115 19.0264 2.45441 18.6849L10.0963 11.0743C10.0963 11.0743 9.3615 9.90338 9.3615 8.29344C9.3615 4.81767 12.1907 2 15.6807 2C19.1708 2 22 4.81767 22 8.29344ZM15.681 10.4889C16.8984 10.4889 17.8853 9.50601 17.8853 8.29353C17.8853 7.08105 16.8984 6.09814 15.681 6.09814C14.4635 6.09814 13.4766 7.08105 13.4766 8.29353C13.4766 9.50601 14.4635 10.4889 15.681 10.4889Z" fill="#1C274C"/>
</svg>
Side by Side

Before

Width:  |  Height:  |  Size: 1.1 KiB

BIN
src/interface/desktop/assets/icons/khoj-logo-sideways-200.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 13 KiB

BIN
src/interface/desktop/assets/icons/khoj-logo-sideways-500.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 29 KiB

BIN
src/interface/desktop/assets/icons/khoj_logo.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 10 KiB

4
src/interface/desktop/assets/icons/link.svg
View File

@@ -1,4 +0,0 @@
<?xml version="1.0" encoding="utf-8"?>
<svg width="800px" height="800px" viewBox="0 0 24 24" fill="none" xmlns="http://www.w3.org/2000/svg">
<path d="M9.16488 17.6505C8.92513 17.8743 8.73958 18.0241 8.54996 18.1336C7.62175 18.6695 6.47816 18.6695 5.54996 18.1336C5.20791 17.9361 4.87912 17.6073 4.22153 16.9498C3.56394 16.2922 3.23514 15.9634 3.03767 15.6213C2.50177 14.6931 2.50177 13.5495 3.03767 12.6213C3.23514 12.2793 3.56394 11.9505 4.22153 11.2929L7.04996 8.46448C7.70755 7.80689 8.03634 7.47809 8.37838 7.28062C9.30659 6.74472 10.4502 6.74472 11.3784 7.28061C11.7204 7.47809 12.0492 7.80689 12.7068 8.46448C13.3644 9.12207 13.6932 9.45086 13.8907 9.7929C14.4266 10.7211 14.4266 11.8647 13.8907 12.7929C13.7812 12.9825 13.6314 13.1681 13.4075 13.4078M10.5919 10.5922C10.368 10.8319 10.2182 11.0175 10.1087 11.2071C9.57284 12.1353 9.57284 13.2789 10.1087 14.2071C10.3062 14.5492 10.635 14.878 11.2926 15.5355C11.9502 16.1931 12.279 16.5219 12.621 16.7194C13.5492 17.2553 14.6928 17.2553 15.621 16.7194C15.9631 16.5219 16.2919 16.1931 16.9495 15.5355L19.7779 12.7071C20.4355 12.0495 20.7643 11.7207 20.9617 11.3787C21.4976 10.4505 21.4976 9.30689 20.9617 8.37869C20.7643 8.03665 20.4355 7.70785 19.7779 7.05026C19.1203 6.39267 18.7915 6.06388 18.4495 5.8664C17.5212 5.3305 16.3777 5.3305 15.4495 5.8664C15.2598 5.97588 15.0743 6.12571 14.8345 6.34955" stroke="#000000" stroke-width="2" stroke-linecap="round"/>
</svg>
Side by Side

Before

Width:  |  Height:  |  Size: 1.4 KiB

1
src/interface/desktop/assets/icons/microphone-solid.svg
View File

@@ -1 +0,0 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 384 512"><!--! Font Awesome Pro 6.4.2 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license (Commercial License) Copyright 2023 Fonticons, Inc. --><path d="M192 0C139 0 96 43 96 96V256c0 53 43 96 96 96s96-43 96-96V96c0-53-43-96-96-96zM64 216c0-13.3-10.7-24-24-24s-24 10.7-24 24v40c0 89.1 66.2 162.7 152 174.4V464H120c-13.3 0-24 10.7-24 24s10.7 24 24 24h72 72c13.3 0 24-10.7 24-24s-10.7-24-24-24H216V430.4c85.8-11.7 152-85.3 152-174.4V216c0-13.3-10.7-24-24-24s-24 10.7-24 24v40c0 70.7-57.3 128-128 128s-128-57.3-128-128V216z"/></svg>
Side by Side

Before

Width:  |  Height:  |  Size: 616 B

Some files were not shown because too many files have changed in this diff Show More