Hi, we are working on the edx Aspects project, where we are using Ralph as our LRS to insert data to Clickhouse. As part of our project we are inserting events from the past, using tracking logs we gathered over the years, and we inserting those events to Clickhouse using Ralph. While doing this we are trying to use the biggest write chunk size of Ralph and biggest batch size of the LMS, to run less inserts to Clickhouse (which is there recommendation). Currently we set the batch size to be 10,000. While doing this, I see in Ralph logs a lot of those:
2025-02-12 09:37:27,540 INFO: 10.35.8.114:11382 - "POST /xAPI/statements HTTP/1.1" 400 Bad Request
Although, I do see past events being inserts to my Clickhouse, as I expect, I don't understand what this message means. I saw in your code, those 400 status codes are meaning some keys were missing from the json events - which makes sense.
But, my question is does it reject all of the batch I have sent (in my case all the 10,000 events) or it just reject the part of the events that were malformed and inserts the rest of the valid events to Clickhouse?
Hi, we are working on the edx Aspects project, where we are using Ralph as our LRS to insert data to Clickhouse. As part of our project we are inserting events from the past, using tracking logs we gathered over the years, and we inserting those events to Clickhouse using Ralph. While doing this we are trying to use the biggest write chunk size of Ralph and biggest batch size of the LMS, to run less inserts to Clickhouse (which is there recommendation). Currently we set the batch size to be 10,000. While doing this, I see in Ralph logs a lot of those:
2025-02-12 09:37:27,540 INFO: 10.35.8.114:11382 - "POST /xAPI/statements HTTP/1.1" 400 Bad RequestAlthough, I do see past events being inserts to my Clickhouse, as I expect, I don't understand what this message means. I saw in your code, those 400 status codes are meaning some keys were missing from the json events - which makes sense.
But, my question is does it reject all of the batch I have sent (in my case all the 10,000 events) or it just reject the part of the events that were malformed and inserts the rest of the valid events to Clickhouse?