Large files in notebook folder slows down nvUltra
Completed
I had around 1.2 GB of PowerPoint files and PDFs laying around in the folder that contains all my nvUltra notes, and it caused tremendous lag when typing or scrolling through a note.
For example:
./Notebook/ - Host folder ~ 1.5 GB
./Notebook/Subject 1/ - Folder that contains three items
./Notebook/Subject 1/Documents/ - Folder that contains ~ 1.2 GB of PowerPoint and PDF files
./Notebook/Subject 1/Images/ - Folder that contains images for Notes.md ~ 300 MB of .PNG files
./Notebook/Subject 1/Notes.md/ - Markdown file for nvUltra
Interestingly, searching for notes or changing to a different note using the search function (⌘+L) did not seem to be affected by the lag. Clearing the index cache did not fix the problem. What worked was removing all the contents in "./Notebook/Documents/" to a different folder unrelated to "./Notebook/," e.g., "./Example Folder/." It should also be noted that I have ~ 4.5 GB worth of notes within "./Notebook/," but did not notice this problem until I started adding PowerPoint and PDF files into the host folder.
-
nvUltra doesn't index PowerPoint, so that has no effect. But indexing 0.6 GB (or whatever) of PDFs will take it a while to get through.
Unfortunately, there seems to be a leak in macOS involving PDFDocument. Indexing new PDF files leaks a bit of memory for each page indexed. I've been unable to find any fix for this online, and it still seems to be present in 10.15.
Clearing the index cache makes the problem worse, since the files now need to be indexed again.
All that said, I just dug through some more and stress tested with a folder full of PDFs. I found a couple of things, and tweaked some behavior. So this should get better in next release.
1) Fixed some of the multi-threading behavior related to PDFs. Because PDFs were so slow, they were done asynchronously, causing the indexing progress bar to move more quickly even though indexing was still occurring. This got the files listed faster, but made it less clear to the user what was happening. Now I do two things -- there is a "fast" indexing that only indexes file paths to populate the notes list quickly, and then the full indexing happens in a second pass. Additionally, PDFs are now indexed inline with the rest of the files. This still happens on a background thread, but all on the same thread. This should limit indexing to a single CPU core, so it should have less impact on the app/system when indexing a folder full of PDFs.
2) I tried some additional tricks to limit memory usage from PDFKit. There still seems to be a leak, or more accurately, a large cache that PDFKit holds onto. It appears that this is released when the folder is closed. I don't think I can do anything about it, but it seems to be better than in the prior version.
3) I fixed an issue that tried to read PDFs as a straight text file in order to extract MultiMarkdown tags and inline text tags. Obviously that did not work well.
The point above still stands however -- clearing caches is useful when you are experiencing a problem, but keep in mind that it causes the cache to be rebuilt the next time you open the folder. With a lot of PDFs, this can be time consuming.
Let me know if the next release is better for you.
Thanks!
0 -
New beta is out -- please let me know if this improves things for you.
Thanks!
0 -
Sorry for not getting back sooner, had the day off and so I wasn't really using nvUltra for producing notes.
Unfortunately, the update has not changed anything. I've even limited the amount of total files within the host note folder to 1.2 GB: PowerPoints (.ppt and .pptx) occupying the most space by far ~ 900 MB, PDFs at ~ 100 MB, image files (.png and .jpg) at ~ 100 MB, HTML files at ~ 50 MB, a combination of Word (.doc and .docx) and Excel (.xls and .xlsx) files at ~ 50 MB, and Markdown files filling up the rest. Oddly enough, nvUltra hasn't been responding at times and I've had to kill the process multiple times this past hour.
0 -
Here is an example of what the lag looks like: https://gfycat.com/foolhardywelldocumentedangelwingmussel.
Backspacing is very laggy and does not register the same as when typing. I can type and it'll show up very late, but when holding backspace, it refuses to do anything and often gets stuck.
0 -
I've since removed all files except images and Markdown files from the host folder, reindexed, and nvUltra is working fine again. I'm guessing that PDF leak is a problem that I'll have to circumvent by reorganizing files until a fix is found, or if macOS 10.16 helps out haha.
0 -
Can you send a video that includes the entire window instead of just a portion? And you are using build 56 right?
0 -
I'll get back to you on the video sometime at the end today P.S.T.
Yes, I was using build 56 in that video.
0 -
How is build 57?
0 -
Sorry about that, I've been pretty busy recently. Testing it out right now and here's what I've got:
https://gfycat.com/joyfulecstaticcusimanse
It hasn't seemed to fix the issue. It also causes nvUltra to close rather slowly
Also, I'm not sure if the caching index bar is supposed to always be flashing, but whenever I type notes now, that is what happens every couple of seconds.
Moving the PDFs out of the host folder did not fix the problem either; I usually have to uninstall and delete all associated files, then reinstall to fix it.
0 -
Found an issue where updated files were being read on the main thread. This should be fixed in next beta.
0 -
Build 58 should improve this yet again.
0 -
Moved everything back into the host folder and everything seems to be running smoothly on 58. I'll report back if anything gets weird again. Thanks!
--
Scratch that ... found a way to get nvUltra to become unresponsive. I'll comment tomorrow after I figure out how to consistently do it.
0 -
Look forward to hearing about the new issue!
0 -
Pretty sure it's an HTML file causing it to crash (no dialogue saying it crashes though, it just becomes unresponsive). I currently cannot open nvUltra anymore and will reinstall in a bit. My CPU usage also spikes up to 50% when it happens (MBP 16" 2019).
Here's a link to the file: https://drive.google.com/open?id=1lfqcfGkzP_CKTzzBtjLFTBO8hZ8r5-Q1
0 -
That's a 31 MB text file. nvUltra's not going to index it instantaneously. It's also a large file to load whenever you select it from the notes list to display in the editor pane (and preview pane if you have that visible), so that will cause a long delay as well.
To make it worse, it's an HTML file with embedded graphics, which are encoded as text. So it seems to be a text file with lots of really long words that need to be indexed. And since nvUltra supports partial word matching (e.g. typing `f-o-o` will match `foo` at each key press), each of those long words is indexed n times. (A 10 letter long word is indexed 10 times, as a 1 letter word, 2 letter word, 3 letter word, etc.). When you have "words" that are each an entire line of text long (or longer), indexing is going to be *really* slow.
I continue to work on improving performance of the various components used in nvUltra, but indexing, loading, and applying syntax highlighting to a 35 MB HTML file is not the use case for nvUltra.
1. You can disable syntax highlighting for files with `.html` extension to at least speed up part of it.
2. You can ignore files with the `.html` extension to keep them out of the notes list to avoid accidentally selecting them
3. Over the years, I have had to experiment back and forth with the `allowsNonContiguousLayout` option for macOS NSLayoutManager's. In the past enabling that has been problematic. I'll need to experiment more, but turning it on does speed up loading the HTML file into the text view, but it is still slow.
4. But the reality is, none of those things will really give satisfactory performance with a 31 MB text file that is structured like this.
As an aside, when the CPU usage is high like that, the app did not crash. It's busy. An important distinction when trying to troubleshoot what is going on. Since in this case the problem is that you have asked it to do something that takes on the order of 10s of seconds to minutes (on my computer). So you have to be patient, OR not ask it to do that. ;)
PS> You are of course welcome to uninstall and reinstall nvUltra. But that's not going to fix anything, so is generally a waste of time unless you have reason to believe that an update was corrupted? With a version purchased from the App Store, it's even less useful since the app is verified at launch. Just hate to have you wasting time doing something that isn't likely to help in this situation
0 -
Am I correct in understanding that the original topic of this thread is now fixed in build 58? I don't want to close it prematurely given how complicated this discussion has been.
0 -
Yeah, I typically don't open HTML files on nvUltra, I just happened to have that inside my notes folder and I thought it was the Markdown file that I was editing.
Also, after opening the HTML file and force quitting nvUltra, I wasn't able to open nvUltra anymore. I waited about an hour, tried again, and also rebooted my system, but neither of the two allowed me to open nvUltra. It would that the process is running, but no windows would open and the CPU usage would be an absurd number (I also left this running for an hour, but still nothing).
Yes, you are correct, build 58 has fixed the original topic of this thread and can be closed. Thanks.
0 -
What sort of computer are you using? RAM? CPU? Solid state vs spinning drive?
On my 2012 Mini with 16 GB RAM and SSD drive, nvUltra indexes that file in a minute or two (ball park, maybe 2-5 minutes) and loads it in 10-20 seconds). So it's slow, but not reboot my system slow.
But either way, that file is the perfect storm of what *NOT* to use with nvUltra for reasons above.
0 -
Model: MacBook Pro (16-inch, 2019)
Processor: 2.6 Ghz 6-Core Intel Core i7
Storage: 512 GB SSD
Memory: 16 GB 2667 DDR4
Graphics: Intel UHD Graphics 630 1536 MB / AMD Radeon Pro 5300M 4GB VRAM
0
Please sign in to leave a comment.
Comments
19 comments