42 Commits

Author	SHA1	Message	Date
Nicolas	e5b797549e	Merge branch 'main' into feat/scrape-monitoring	2024-07-25 16:21:02 -04:00
rafaelsideguide	309728a482	updated logs	2024-07-25 09:48:06 -03:00
Gergo Moricz	d57dbbd0c6	fix: add jobId for scrape	2024-07-24 15:18:12 +02:00
Gergo Moricz	7cd9bf92e3	feat: scrape event logging to DB	2024-07-24 14:31:25 +02:00
Nicolas	30e706b43f	Update scrape.ts	2024-07-22 19:15:24 -04:00
Nicolas	575ddc9e6e	Update scrape.ts	2024-07-22 19:12:51 -04:00
Nicolas	e31a5007d5	Nick: speed improvements	2024-07-22 18:30:58 -04:00
Nicolas	2fab2d8d29	Update scrape.ts	2024-07-17 20:44:34 -04:00
Eric Ciarla	e1af815f8c	Update scrape.ts	2024-07-01 08:48:21 -04:00
Eric Ciarla	87b54488d3	update to includeRawHtml	2024-06-28 17:07:47 -04:00
Eric Ciarla	70fcf2ce03	init	2024-06-28 16:39:09 -04:00
rafaelsideguide	4381109dd8	added default values and fixed pdf bug	2024-06-26 09:00:54 -03:00
Rafael Miller	f9c7ca9388	Merge branch 'main' into feat/issue-266	2024-06-14 11:47:58 -03:00
rafaelsideguide	5dd18ca79b	fixed edge cases	2024-06-14 09:46:55 -03:00
rafaelsideguide	e37d151404	added parsePDF option to pageOptions ... user can decide if they are going to let us take care of the parse or they are going to parse the pdf by themselves	2024-06-12 15:06:47 -03:00
Nicolas	5e80f8af87	Nick: llm extract 50	2024-06-06 18:35:44 -07:00
Nicolas	0c115c6181	Merge pull request #216 from mendableai/nsc/new-pricing ... feat: New pricing/limits changes	2024-05-30 15:36:59 -07:00
Nicolas	6ceb7ff50a	Nick:	2024-05-30 14:46:55 -07:00
Nicolas	ace46f340b	Nick: new limits, new pricing	2024-05-30 14:31:36 -07:00
Eric Ciarla	a0e404f94e	init commit	2024-05-29 18:56:57 -04:00
Nicolas	1b3547dcf2	Nick:	2024-05-28 12:56:24 -07:00
Nicolas	77a79b5a79	Nick: max num tokens for llm extract (for now) + slice the max	2024-05-20 17:07:38 -07:00
Nicolas	65d89afba9	Nick:	2024-05-13 13:01:43 -07:00
rafaelsideguide	e1f52c538f	nested includeHtml inside pageOptions	2024-05-07 13:40:24 -03:00
rafaelsideguide	509250c4ef	changed to includeHtml	2024-05-06 19:45:56 -03:00
rafaelsideguide	538355f1af	Added toMarkdown option	2024-05-06 11:36:44 -03:00
Nicolas	a386259511	Update scrape.ts	2024-04-30 16:35:44 -07:00
Nicolas	dfcf39f4c0	Update scrape.ts	2024-04-30 16:19:59 -07:00
Nicolas	4f526cff92	Nick: cleanup	2024-04-30 12:19:43 -07:00
Caleb Peffer	d9d206aff6	Caleb:	2024-04-30 10:27:39 -07:00
Caleb Peffer	3ca9e5153f	Caleb: trying to get loggin workng	2024-04-30 09:20:15 -07:00
Caleb Peffer	4f7737c922	Caleb: added ajv json schema validation.	2024-04-29 12:12:55 -07:00
Caleb Peffer	06497729e2	Caleb: got it to a testable state I believe	2024-04-28 15:52:09 -07:00
rafaelsideguide	06675d1fe3	almost finished	2024-04-26 11:42:49 -03:00
Nicolas	f3c190c21c	Nick:	2024-04-23 16:47:24 -07:00
rafaelsideguide	849c0b6ebf	[Feat] Added blocklist for social media urls	2024-04-23 18:50:35 -03:00
Nicolas	5cdbf3a0ac	Nick: cleaner functions to handle authenticated requests that dont require ifs everywhere	2024-04-21 10:36:48 -07:00
Caleb Peffer	be75aaa195	Caleb: first version of supabase proxy to make db authentication optional	2024-04-21 09:31:22 -07:00
Nicolas	0db0874b00	Nick:	2024-04-20 19:37:45 -07:00
Nicolas	4543c57e4e	Nick:	2024-04-20 19:04:27 -07:00
Nicolas	5b8aed26dd	Update scrape.ts	2024-04-20 18:55:39 -07:00
Nicolas	23b2190e5d	Nick:	2024-04-20 16:38:05 -07:00