Try fixing non-US stock charts

[ymcx]

Continuation of #186 and #184.

Stuff that might need to be taken care of before we can merge:
1. Account for holidays
2. Make it work for market times that span from the previous day to the next day
3. Fix non-1D low resolution US stock charts
4. Verify the correctness (including volume & date labels), or if this approach is working at all
5. Make the date labels use this new approach for non-1D stocks as well
6. Crypto
7. DST
8. Gaps
9. Fix volume alignment
10. DEFAULT_TIMESTAMPS

I'm unsure if this is what you envisioned, but it seems to be working all right (excluding US stocks for whatever reason).

[tarkah]

Hey thanks! Yeah this is pretty much what I envisioned for generating a discrete axis for each stock based on its market times. I don't expect us to be 100% accurate and I think that's okay for this application, but we should still strive to tackle known gaps that we have the ability to identify & correct. Holidays are probably out of scope if we can't get them.

This seems to assume market open / close fall on the same UTC day, so that's one change we need to potentially address. We also don't account for DST adjustments in our seek logic at week end to week start.

Also, we probably don't want to do this for is_crypto charts since they're continuous and there should be no gaps in data, so we can just naively return all price data for them. Ideally we use actual continuous x-axis based on timestamp but that's a bigger refactor that changes how we define the chart x-bounds & dataset x point so I wouldn't scope that out under here.

Lastly it seems like our market start - timeframe adjusted start time is off, as it includes the partial previous day to the actual start of the (1W == 5 full business days), giving a small to big gap on the left edge of the non-1D charts. We should saturate the adjusted start on the start of the first full day. We can do this by seeking to the start of the next business day start time first, then seeking back the time_frame from there.

[ymcx]

Alright, while your suggestion fixed the small gap in 1W, 1M still contains a gap. INTC:

[ymcx]

Do you have any idea what could cause this smoothness? Here's BTC-USD, but the exact same thing happens with INTC

[ymcx]

nvm I think I fixed it apparently not

[tarkah]

A few things I'm noticing in testing between master and this:

• I'm not seeing the smoothing behavior you mentioned for INTC or BTC-USD, did you fix this?
• % change stat is different, meaning the first value of the series is being captured differently. Is this more or less accurate than before? Maybe we should use the raw data & not the computed prices array.
• Toggling pre/post changes the chart on non-1D timeframe when it shouldn't since we only fetch pre-post data for 1D. start_end should probably only return pre_start and post_end for timeframe 1D otherwise it returns regular market times.
• 1M+ timeframes are messed up since we're seeking to next business day prior to subtracting months / years from the date. I think we need to subtract from the current day for these timeframes, then probably seek to the next business day from there if we land on a weekend. I'm unsure exactly how yahoo handles these ranges but that seems to fix them. It seems they might subtract num_days_in_prev_month from current day for 1M but for 1+ month it seems like they do next_day - 30*num_months. 😅

Edit: Can we still use SPY / DEFAULT_TIMESTAMPS as a proxy for 1W+ adjusted start date (then add market aware time to it) so we're always aligned with how Yahoo specifies its ranges?

[ymcx]

I'm not seeing the smoothing behavior you mentioned for INTC or BTC-USD, did you fix this?

Can't reproduce it either anymore. I honestly don't remember if I fixed it or not. Thankfully it's easy to notice if it reappears.

% change stat is different, meaning the first value of the series is being captured differently. Is this more or less accurate than before? Maybe we should use the raw data & not the computed prices array.

Can confirm, the old one is more accurate. Using the raw values is absolutely the way to go here, I'll do that.

Edit: Can we still use SPY / DEFAULT_TIMESTAMPS as a proxy for 1W+ adjusted start date (then add market aware time to it) so we're always aligned with how Yahoo specifies its ranges?

Do you mean we should use both DEFAULT_TIMESTAMPS and MarketTimes (prev. MarketHours) so that the dataset items may align to either of those in order to be accepted?

[tarkah]

Do you mean we should use both DEFAULT_TIMESTAMPS and MarketTimes (prev. MarketHours) so that the dataset items may align to either of those in order to be accepted?

I just mean we use it as a proxy for the adjusted start that is passed into MarketTimes, but we still use MarketTimes. Because landing on an accurate adjusted start implementation seems like it'll be difficult without understanding exactly how Yahoo returns data for each range / interval combination.

[ymcx]

subtract from the current day for these timeframes, then probably seek to the next business day from there if we land on a weekend

That still produces quite large borders with 1W and 1M. 1W goes from Friday 5/1 to Friday 4/24, which is not a weekend. For some reason 4/24 is completely empty.

[tarkah]

subtract from the current day for these timeframes, then probably seek to the next business day from there if we land on a weekend

That still produces quite large borders with 1W and 1M. 1W goes from Friday 5/1 to Friday 4/24, which is not a weekend. For some reason 4/24 is completely empty.

Right, this will be difficult to get right which is why I suggest we proxy start off SPY and run the market times fill from there

[ymcx]

proxy start off SPY and run the market times fill from there

Aight, I'll look into this tomorrow since it's past 1AM here :D

[tarkah]

proxy start off SPY and run the market times fill from there

Aight, I'll look into this tomorrow since it's past 1AM here :D

Gn! Thanks for tackling this, it's coming along nicely.

[ymcx]

After SPY proxy there are still (at least) two things that should be addressed:

Is there anything we can do to account for DST? I'm not exactly sure how it works currently and it's not that trivial to test it.

What about the volume alignment issue? A quick git bisect led me to the initial commit from this PR, but other than that I can't seem to find the root cause. This is even though it uses the same normalized values as the price chart and the date labels.

[ymcx]

Sorry if I'm making too many changes, especially those that aren't directly related to the original goal of this PR. I just decided to bundle them all up into this one PR so I don't have to create a bunch of new ones again just for some trivial changes.

I moved the help 'button' into the Toggle block, since it had one empty slot and IMO it doesn't make much sense to have a similar button be placed elsewhere. Now the button is hidden in the summary view, but I figured that since the summary view is opt-in anyway, the user has already either toggled the summary view using the CLI argument or alternatively has already taken a look at the Toggle block in order to be able to switch to the summary view. I also removed --hide-help since it's not needed anymore.

Please do mention if you don't like some of these so that they can be reverted.

[ymcx]

I changed the keybinds up a bit. The primary change is that now tab/shift+tab modifies the time frame whereas the arrow keys change the selected stock. I made this change to unify how stock changing and swapping works. Before you would change the stock selection using tab and swap them using the arrow keys with a modifier. Now both are done with the arrow keys.

I would've kept having tab as the way to change stocks, but there aren't any modifiers left to apply for the swap action after having assigned shift+tab already.

[ymcx]

I seriously don't know why the Windows test workflow is failing. Like I traced it down to isahc, and the issue started once I updated the dependencies with isahc from 1.7.2 to 1.8.1. I tried in another repo with 1.7.2 and it's still failing.

Turns out it was curl-sys which was updated while updating the lock file

[tarkah]

@ymcx nice work w/ all the new changes. Can you please split all of those commits into a new branch / PR? I'm fine if they're all bundled together, but I'd prefer they existed separate from the x-axis work you're doing since that's a bigger change & I want to keep it isolated for review & testing purposes.

Also, what's the latest status on the x-axis changes? Are there still pending changes, such as the SPY adjusted start proxy value?

[ymcx]

Can you please split all of those commits into a new branch / PR?

Sure, I'll open a new PR with all of these changes if that's okay. I kind of lost myself in just implementing new features and now this PR has became much bigger than what I anticipated. After all these are some of my first years programming, so I'm still somewhat trying to figure out how things should be done in a manner that is expected from us.

Also, what's the latest status on the x-axis changes

Haven't yet been able to really focus on that since I figured I would first do the cosmetic changes and stuff before getting back to fixing the initial issue. But as you can see, that hasn't worked out so well since I constantly had new ideas that got the priority.

[ymcx]

I created another branch for this bugfix, you can find it here https://github.com/ymcx/tickrs/tree/fix-charts. I'll perhaps make a new PR (again) once it's done?

Anyway, with your suggestion of using DEFAULT_TIMESTAMPS as a proxy for 1M and up, INTC works flawlessly without any kind of gaps or smoothness. As for NOVO-B.CO, from 1M onwards it starts getting both more smooth and the same gaps appear that were there without DEFAULT_TIMESTAMPS.

[ymcx]

Can't we just create a new DefaultTimestampService for each stock? I tried that and not only is the result literally perfect, but it also requires much less code.

[tarkah]

Can't we just create a new DefaultTimestampService for each stock? I tried that and not only is the result literally perfect, but it also requires much less code.

What does this look like?

[ymcx]

What does this look like?

As good as every NYSE stock did, since after all, we're using the exact same method, now just for each stock separately

[ymcx]

I'll submit this as a standalone PR since this is much better as long as it's fine if we fetch the default times for each stock separately. There isn't really any other reason not to.

[ymcx]

I figured this branch would stay open w/ just the commits related to it

yeah but the new branch only contains three commits, all of which don't require the commits in this branch so I figured I'd just create another branch with all of the unnecessary changes not applied. The fix doesn't even depend on the earlier commits in this branch which is why I started from HEAD.

I'm unsure if it's possible to just start from scratch in this branch without polluting the commit history too much but I'll try. Nevermind, it worked wonderfully

I'm sorry for all of the headache this has caused :D

[tarkah]

I'm sorry for all of the headache this has caused :D

No worries at all :) Yeah you can always just force push to a branch to rewrite its history. I'd highly recommend checking out jj, its a much more natural workflow for dealing w/ change history and easily squashing, rebasing, moving history around / etc. gits mental model is a bit harder to internalize and much harder to manipulate so freely.

[tarkah]

Since we're trying to match Yahoo's times exactly for each stock separately, yet our only DefaultTimestampService was built for SPY and nothing else. As such it perfectly follows the market opening/closing times for NYSE, but nothing else that's in a different timezone.
So as for the implementation, it's very simple in terms of amount of lines changed. I only needed to remove the app's main DefaultTimestampService and instead create one for each StockState. Upon the creation of a StockState object, we're then initializing a new DefaultTimestampService with the symbol of the stock, which then returns the correct "default timestamps" for that stock.

We already have the timestamps for each stock though. The issue we're trying to solve is an accurate adjusted start time that is TimeFrame before now. If that stock wasn't trading then, these values wont exist. Hence we use SPY as the proxy. We don't care about the start time of the first SPY timestamp. We just care about the date rounded to midnight, then we can add the start_time based on the stocks regular market hours. This should be good enough?

@ymcx what did you think about this?

[ymcx]

I'd highly recommend checking out jj

I think I've actually heard about jj on HN, but never bothered to check it out as git is the industry standard (I know that it's somehow compatible with git). Now that you piqued my interest, I'll absolutely take the time to actually try it out someday.

This should be good enough?

I'm not totally sure on how the timestamps are currently fetched based on the given ticker (SPY in this case), but to me it seems like we're getting the market opening and closing time for the stock in question and turning it into a time series using a delta, correct? And basing off of that, it seems reasonable to fetch the start and end times (via default_timestamps) for each stock (/ stock exchange) separately, as that will take care of timezone differences, the actual amount of time the market will be open and probably even the holidays?

Running a background service for something so apparently trivial seems unnecessary, but I think it's cleaner to do it this way in contrast to having to deal with custom deltas that need to be added to the timestamps of NYSE. The last approach involved a lot of hacky code just so we could correctly handle all of the edge cases and it still felt brittle with those pesky gaps and smoothing effect appearing sometimes, for no apparent reason.

I'm cool with doing it that way, but I'm out of ideas on how we would actually make it work. I'm all ears if come up with a clever solution.

[tarkah]

I'm still not sure I follow how running DefaultTimestamps for each stock individually solves anything? All DefaultTimestamps task does is fetch chart data and output all timestamps for each timeframe. We already have this data for each stock, so we're not fetching any new data in that case. If a stock was listed 1 year ago, then on 5Y we wont have a 5y ago timestamp as its first timestamp. That's the entire reason we would instead proxy off SPY since SPY has an accurate enough first timestamp for each time frame.

I agree its wasteful to run this additional task, but we already know theres some issues in trying to manually project out the adjusted start time based on NOW - time_frame because we don't know enough about how yahoo is returning these ranges based off interval&range query params.

If we use SPY as the proxy, it should be close enough and probably closer than our manual adjusted logic. All we need to do is normalize that first SPY timestamp to the current stocks timezone / regular market start_time and our MarketTimes generator should produce a usable discrete axis for us to normalize data against.

We already incur the cost of this task today, so its not a regression in performance. Ideally we can get rid of it, but without a known, pure accurate adjusted start time computation, I think its the best we have.

[ymcx]

I put the following code in prices() in src/widget/stock.rs:

...
} else if let Some(default_timestamps) = default_timestamps {
    let f = default_timestamps.first().unwrap();
    let date = NaiveDateTime::from_timestamp_opt(*f, 0);
    log::info!("{:?}", date);
    default_timestamps
...

Now when I open tickrs using a time frame of one week and switch between stocks that are not on the same stock exchange, something like this gets appended to the logs:

[00:00:06.895] (7f4b48eb86c0) INFO   Some(2026-04-30T13:30:00)
[00:00:07.155] (7f4b48eb86c0) INFO   Some(2026-04-30T07:00:00)

The goal of this is to make non-US stocks work just as well as US stocks do. Currently we have timestamps that follow NYSE and when there is a stock that isn't on NYSE, i.e. its market opens at a different time, we're discarding everything that doesn't fall within the NYSE window. This is why NOVO-B.CO on the Copenhagen stock exchange was cut off to just a couple of hours of data each day with the rest being a straight line to the next item on the next day when viewing on 1W.

So if the problem is that earlier we had to choose which exchange to follow (we chose NYSE via SPY), we would be able to solve the problem by calculating the same for each stock or more specifically each exchange or market start/end time.

All we need to do is normalize that first SPY timestamp to the current stocks timezone / regular market start_time and our MarketTimes generator should produce a usable discrete axis for us to normalize data against

Exactly, but now the problem becomes how do we get that time delta? You're correct in saying that this doesn't help much in the sense that we're just fetching the time delta between whatever timezone NYSE uses and the timezone of the stock exchange of the current stock. I figured that this would be the easiest approach to gather that data without the need of perhaps hard-coding the time deltas of each stock exchange and then also trying to somehow deal with DST (I'm not sure if it works OOTB when using Unix epoch).

If I'm understanding you correctly, the biggest issue you have with this approach is that we're wasting Yahoo's resources by pinging them more often than absolutely necessary, then yes, I agree. What do you think, would it be better to instead try to calculate the time delta ourselves using some hardcoded data about the exchanges?

[tarkah]

Exactly, but now the problem becomes how do we get that time delta?

Can't we use chart_meta.current_trading_period.regular.start? So we use SPY proxy first timestamp -> round midnight + chart_meta.current_trading_period.regular.start.

We can also do a min between the charts actual start timestamp and this calculation to correct slight inconsistencies where the chart does span the full period and this was somehow still incorrect (potentially when regular.start > regular.end and it actually started on the previous day).

If regular.start isn't available, we can either just use SPY as is, midnight as is, or we can calculate the min start time from midnight across all timestamps of the chart for that time frame. This data should hopefully mostly be available so not even sure if this is something we will run across or not.

Considering all of that, we should have something very accurate or accurate enough for our purposes.

[ymcx]

Can't we use chart_meta.current_trading_period.regular.start?

I didn't realize we already had the market opening time. In that case, we can indeed use it to calculate the exact same thing. I'll update the PR to use this approach instead.

[ymcx]

I tried that approach out and perhaps unsurprisingly, it worked well with the exception of holidays. We're after all presuming that on each day the market is open at the exact same time as long as we add the static delta, but that's hardly ever the case. I'm not sure about the US, but a large part of Europe celebrates Labor day, which was on the 1st of May. NYSE seems to have been open during that time, but for example Xetra (Frankfurt Stock Exchange) wasn't.

Because of this we might get artifacts like this:

On the right you can see the current behavior when compiling from this branch and on the left there is the one where the default timestamps were used with a delta.

[tarkah]

That makes sense. Instead of fitting the data to a generated discrete axis over the entire time frame, should we just sandwich the actual chart data between two projected axes? Those projected axes use the MarketTimes iterator and on the happy path will be empty if the actual chart data spans the entire time frame. For stocks where they started / stopped trading somewhere within the timeframe, it'll fill in that gap.

adjusted_start..start | start..end | end..adjusted_end

Then we'll never show the holiday gaps within the actual chart data since Yahoo doesn't produce data for those.

Wdyt?

[ymcx]

I'm having a hard time wrapping my head around what exactly you're proposing. Do you mean we should use the new approach with default_timestamps + the delta and then add a fallback, MarketTimes, which will catch all items that didn't match default_timestamps due to differences in market hours compared to NYSE?

[tarkah]

Ok, I think we need to align on a few things, especially because you rolled back the MarketTimes change we we're originally aligning around to something else.

On this current branch, you fetching DefaultTimestamps for each stock and using that is identical to just plotting the chart data directly, since you're fetching the same underlying timestamps as we already have in self.prices. If you delete the whole if let Some(default_timestamps) = default_timestamps block and therefore just use prices directly, you'll get an identical result. So effectively the fix this branch currently has is just... plot things directly. The issue is that this stretches the chart data over the entire x-axis and stocks like ASST which don't span 5Y get stretched out / don't show the gap.

I'm suggesting we use the MarketTimes implementation you had previously, but ONLY using it before & after the actual self.prices data to extend to the start of the time frame & end of the time frame. And we use SPY w/ DefaultTimestamps (as master) does to have a proxy start / end time for each timeframe (adjusted to the current stocks regular market start / end hours). The effect being this:

| <-- fill --> | <-- prices --> | <-- fill --> |
 \              \                \              \  
  \ (adj_start)  \ (start)        \ (end)        \ (adj_end)

This has the benefit of:

• Showing accurate graph where we have data since we plot that data as is
• Adding start / end padding if the stock didn't trade the entire time frame

The rust code would be something like:

let (adj_start, adj_end) = self.adjusted_start_end(prices);
let delta = self.time_frame.round_by();

let start = prices.first().unwrap().date;
let end = prices.last().unwrap().date;

let fill_before = MarketTimes::new(adj_start, start, delta);
let fill_after = MarketTimes::new(end, adj_end, delta);

fill_before
    .map(Price::new)
    .chain(
        prices
            .iter()
            // Needed to filter out pre / post market data if pre / post is disabled
            .filter(|p| p.date >= adj_start && p.date < adj_end)
            .copied(),
    )
    .chain(fill_after.map(Price::new))
    .collect()

[ymcx]

Aight I think I understood. I'll see how it goes and report back either tomorrow or today if the the change is trivial.

[tarkah]

Sounds good. Note I have a branch with this basically mocked up. Let me know if you want me to push it as a reference branch. Its based on your older commit history.

Edit: Checkout fix-non-us-charts-tarkah, specifically 723f63e. It builds off your older commits but just makes the proposed change above. I've tested this and it seems to work really well. I'd like to test it during regular market times next week.

[ymcx]

This is what I came up with. Even though our implementations are not completely the same, it seems like they achieve the same end result.

1D is a little broken so I'll fix it later

[ymcx]

Nevermind, my approach is completely borked not only in 1D but also when padding it somehow divides the padding incorrectly. IMO we should just go with your approach since it seems to be working anyway. Will you merge your branch and then I'll go and close this PR for good?

[tarkah]

Sounds good. I'll force push that history to this branch so we don't lose all the comment history here and then I can merge.

[tarkah]

Ok, I've pushed. Is there anything else we need to test?

I noticed there is still a small gap on 1M chart and it appears to be a slight discrepancy between the first data point returned being later than the regular market open times by 1 hour for NYSE and 7.5 hours for european exchanges. No idea why Yahoo has this offset for 1M but no other time frame. And no idea if this delta will always show up for 1M or its just a spurious issue today.

Its hard for me to recommend a fix without knowing the behavior of Yahoo's API and if this will always be a problem or if its spurious.

Edit: As I'm typing this, NOVO-B.CO is now correctly coming across with its first timestamp at the exact start of the trading period. So this seems like a spurious issue and I don't think we try to solve for it. Any further fix to address this will increase complexity and isn't guaranteed to work. In the grand scheme, this is a very minor issue compared to everything we've fixed.

[ymcx]

Is there anything else we need to test?

I can't think of anything that we haven't discussed here already. Have you verified that holidays work correctly? X labels worked last time I checked, but what about volume bars? It's not like we touched them at all, but they used to be misaligned some commits ago.

this is a very minor issue compared to everything we've fixed.

Agreed. IMO we should just accept that the chart can't be rendered perfectly (or at least not with the current approach) and move on. Later down the line there might be reason to come back to this and really dive deep into why we have these deviations compared to Yahoo Finance's official charts etc.

[ymcx]

I don't mean to rush you, but will you be merging this? I'm waiting until this has been merged before submitting the keybind refactor so that there would be no merge conflicts.

[tarkah]

I don't mean to rush you, but will you be merging this? I'm waiting until this has been merged before submitting the keybind refactor so that there would be no merge conflicts.

Merging now 👍 Sorry I lost track of this