NetBeans IDE is scanning/parsing/indexing for ages? Yeah, it's a bug!

"Scanning in progress..." - I believe that everyone have met the issue. You open some larger project and you can not use the NetBeans IDE for a few minutes until the sources are "scanned", whatever it means...

Some unsorted facts first

  • scanning is needed and must be performed, otherwise features that need a knowledge of the source code structure (code completion, refactoring, go to ...) could not be used
  • only the first scanning should be slow - the result is cached in your userdir ($HOME/.netbeans/$VERSION) and further parsing is incremental (only the part that is modified is scanned again)
  • editor infrastructure is very new - Parsing API was introduced in 6.5 for Java and in 6.7 for other languages, therefore, some optimizations are not made yet (for example scanning files in a more intelligent order, allowing to use some features while scanning, etc.). Scanning itself is absolutely no problem - the problem is that the IDE is not very usable while scanning is in progress...
  • scanning&indexing issues are the most commonly heard "deal-breakers" for the Netbeans IDE and the NetBeans dev team knows about them

Submitting scanning&indexing issues

If you are in the situation when the scanning&indexing takes too much time or if the IDE freezes during scanning a project, it is an issue. The only way to increase the chance someone is able to fix it is to submit the issue and to have enough patience to handle the issue (reply questions, provide more details, don't get offended by developer's questions/comments...).

Let's summarize what you need to do in order to file a useful bug report on scanning:

  1. In order to improve logging of the scanning related issues, please run IDE with a command-line switch that improves logging:
    -J-Dorg.netbeans.modules.parsing.impl.indexing.RepositoryUpdater.level=FINE
  2. Include the version of NetBeans, Java and OS in you report (copy information from Help>About) and attach IDE log file ($HOME/.netbeans/$VERSION/var/log/messages.log)
  3. Describe your problem as precisely as you can but please try to be brief too (these requirements are contradictory). Specifically for scanning issues, include the answers to following questions:
    • Does the scanning ever finish?
    • If yes, how long did it take?
    • Was it only the first scan issue, or is the problem met whenever you open the project?
    • If scanning didn't finish, did IDE freeze so that it needs to be killed?
    • If yes, did the main window repaint (was AWT blocked)?
  4. Provide as much information about your project's setup as you can - we cannot reproduce all bugs as we don't have every possible setup:
    • What is your project type?
    • How big is your project (number of classes, disk size, ...)?
    • Does your project use some libraries (how many)?
    • What is the structure of your project (i.e.: long "string of dependencies"...)?
    • Is your project public, so that everyone can get the sources and open it?
  5. It very often happens that the performance issues are a result of some specific setup and they can be unrelated to the place where they actually occur - we need to identify possible external cause of the problem:
    • Do you use any version control system for your project?
    • Is your project located on the network drive?
  6. When you meet some performance problem and you intend to submit an issue, please don't forget to capture a thread dump or a profiler snapshot so that we can readily see where the activity is.
These are the points that I see as the most important when submitting the scanning and indexing issues. If you think of some other information that can be useful, please add your comment/hint to the discussion...
Comments:

You already note the biggest single issue above: the IDE is not very useful while scanning. It is essentially an overweight text editor until scanning is done being incapable of Go-To-Type, Go-To-Declaration, or most other Go-To-\* navigation and even incapable of Attach-Debugger!

I do understand that scanning an immense body of code like that which I shove at NetBeans takes time. While I hope the NB team continues to optimize performance in this area, this is never going to be fast enough for arbitrarily large code sets. It is critical that NB be able to proceed with navigation, etc, based on (1) data from the scan so far and (2) data from the previous scan.

Note that re-scan/indexing is not very fast in many cases for me. It is faster, but not fast.

The other, lower-priority issue: navigation is sometimes rather slow, e.g. Go-To-Declaration should be all but instantaneous. This is important to keep an eye on, but the big issue is that which you already noted: the IDE must not become a crippled boat anchor during scanning!

Posted by Jess Holle on June 20, 2009 at 01:04 AM PDT #

Note that scanning sometimes starts up when you'd least expect it \*and\* NetBeans forgets to tell you via the status bar and only tells you when you're attempting an operation which is disabled during scanning.

This is extremely frustrating. Yes, I should file a bug, but it does not happen "on demand", but rather seemingly at random, and I neglected to capture a stack dump, etc, when this last occurred.

[P.S. My projects are large free-form projects interconnected with a large build system and thus not sharable.]

Posted by Jess Holle on June 20, 2009 at 01:09 AM PDT #

"Include the version of NetBeans, Java and OS in you report (copy information from Help>About) and attach IDE log file ($HOME/.netbeans/$VERSION/var/log/messages.log)"

Isn't all of that information in the log file? The easier it is to collect meaningful data, the more data you will get. Let's automate as much as we can. Fir example why not set the logging level to FINE by default in the non-release builds? Daily builds are updated frequently so the less tweaking to the configuration we have to do, the less tweaking we will forget to do :-)

"What is your project type?
How big is your project (number of classes, disk size, ...)?
Does your project use some libraries (how many)?
What is the structure of your project (i.e.: long "string of dependencies"...)?"

The IDE should know all of this, put enough info in the logs (at FINE level anyway).

"Do you use any version control system for your project?"
If it is supported in NB, then the IDE should already know this and it should be in the logs.

Add a bug-reporting mechanism into the IDE so that all of the data is collected automatically - you are already doing things like this for exception reporting. You've already added ability to get profiler snapshots of the IDE... Add one item "Submit problem report" with the ability to automatically grab as much info as it can (being careful to disclose any IP that might go with the report)

Once such a system is in place the feedback is sure to increase substantially - for all problems.

Posted by swpalmer on June 24, 2009 at 12:09 AM PDT #

swpalmer: Thanx for those nice suggestions, all of them are definitely worth considering.

Anyway, it is not in 6.7 yet, so currently you need to do the steps (IDE does not do it). Actually - you might be able to avoid some of the steps I have mentioned... if you capture a thread dump, it can exhibit an activity in the subversion module, for example (which tells us you use svn).

But generally it is better to include as much information in the text description as possible - for a human, it is easier to read a human word than some disgusting log file. If I read "I use subversion for my project", I look for possible subversion activity in the log/dump files to see if it can be related... if you don't write this information, I have just some log file - so I need to ask you a question if you can reproduce the problem with disabled svn, for example...

The similar thing holds for:

> Isn't all of that information in the log file?

Yes, it is, so this is definitely redundant but it is much easier to see this info directly in the text description. Usually when there is some "well known bug", we know exactly at what build it was fixed or that the issue is JDK/OS specific. Providing both information (help>about+log) doesn't cost too much and simplifies reading the issues in the issue tracker...

> For example why not set the logging level to FINE by default in the non-release builds?

For which module? I mentioned FINE logging only for a module related to scanning... If we do it for all modules, the log will be useless because it will not be possible to upload it due to it's size... we would have to ask you to grep the useful information from it - asking you to use one specific switch is easier;)...

Last but not least - it we implement such a system, it won't be bugless and it costs some development time. I am not able to evaluate if it is worth collecting too many details - \*there has to be some balance with this.\*

Actually, there is one more thing to mention... Some of you might have noticed the "System slowness detected" bubble (automatic profiling when AWT thread is blocked). Jindrich Sedek is preparing an automatic upload of profiler snapshots captured with this system, see: http://wiki.netbeans.org/FitnessViaPostMortemAutoReport

Hopefully, we will be able to deliver some performance improvements for 6.8 - it is already in our plan: http://wiki.netbeans.org/EditorPlan68

Posted by guest on June 24, 2009 at 12:32 AM PDT #

I don't mean to pile on here, but is there really a need for more bug reports for this? Searching issuezilla for "task scanning" already shows six issues, which mostly cover the same thing (to wit, that the IDE is essentially unusable while task scanning is going on). I'm going to add in a bug for my own particular issue, which is that if you select "open project" while task scanning is active the whole editor completely locks up, but I'm wondering if it's really all that helpful to have a bunch of issues which will just wind up marked as duplicates of another issue. Going by my own experience, it's not like it's at all difficult to reproduce this, all one needs to do is open a large project for the first time. In my own case, NB has been quietly ignoring me for at least 30 minutes now, more than enough time for me to fire up Eclipse and actually get some work done.

Posted by tim on June 24, 2009 at 03:43 AM PDT #

Hi Tim,

this is a question that sometimes occurs and that sounds logical. My reasoning goes as follows:

The scanning takes long -> it means it is an issue. Issues you meet should be submitted if they are to be resolved... There can be 3 cases when asking a question "Is the bug you are facing already submitted?":

\* Yes - Vote for it and attach comment and any data that can be useful (requested info for the incomplete issues, for example)
\* No - Submit a new issue with data mentioned above
\* I don't know - Submit a new issue anyway, the worst thing that happens is that it is evaluated by dev/QE and marked as a duplicate (or you submit a new bug)

Are you sure that the issue you are facing is the same as the issues that are submitted? There can be many issues saying "scanning is slow" but these can have many different causes...

Also, issue's "real priority" is given by its priority (P1 issues must be fixed), number of votes (reason obvious) and by the number of duplicates (a P3 issue with 140 duplicates gets fixed sooner than a P3 issue with 3 duplicates). Of course it is always better to vote for an issue rather than submit a duplicate one but not everyone is able to evaluate the issue based on - for example - the thread dump... Developer should be able to do it...

This post was about "what meaningful data should be provided in order to make a report of a scanning issue useful to us"... It has been also placed on NB wiki (Users FAQ>Bug Submitting).

Once again a list of improvements that are planned for 6.8:

http://wiki.netbeans.org/EditorPlan68

With regards,

--pd

Posted by Petr Dvorak on June 24, 2009 at 09:25 AM PDT #

Maybe one more tip for the brave ones:

http://wiki.netbeans.org/ScanOnDemand

--pd

Posted by Petr Dvorak on June 24, 2009 at 09:29 AM PDT #

Thanks for the clarification, and good luck with the bug. FWIW, reading your post did cause me to get a stack dump of NB from jstack and submit it with my bug report before I killed the process. Cheers!

Posted by tim on June 24, 2009 at 09:32 AM PDT #

You are welcome Tim, actually - I should thank you for submitting the issue and attaching the jstack dump. :)

Posted by Petr Dvorak on June 24, 2009 at 09:46 AM PDT #

> Parsing API was introduced in 6.5 for Java

AFAIK Parsing API was not introduce in 6.5, but in 6.7

Posted by Jan Becicka on June 25, 2009 at 09:54 PM PDT #

Jan, thank you for clarification (I talked to Jan offline) - the Parsing API was implemented for Java in 6.5 (in a special repo) but it was not merged in 6.5 release (as it was only for Java, we decided to wait till 6.7 with the merge)...

Posted by Petr Dvorak on June 25, 2009 at 11:29 PM PDT #

Is this also related to the automatic error checking that seems to be driving everybody nuts with the red-wavy-lines and error icons ???

It's very distracting and annoying to be reminded of the missing semi-colon even before you reach halfway our first line of code.

Zinahe A.

Posted by Zinahe on July 23, 2009 at 01:39 AM PDT #

Zinahe: Not really. Basically, the scanning&indexing has to be done anyway. Yes, the results are used for the error checking too, but the scanning is not done only because of this (I believe - don't know - the red underline works in the similar way the syntax highlighting does)...

Anyway, here is the issue:

http://www.netbeans.org/issues/show_bug.cgi?id=169067

you can help it by voting for it...

Posted by Petr Dvorak on July 23, 2009 at 04:33 AM PDT #

"only the first scanning should be slow - the result is cached in your userdir ($HOME/.netbeans/$VERSION) and further parsing is incremental (only the part that is modified is scanned again)"

Today as I was using NB 6.7 I had to wait several times for scanning to complete before ctrl-clicking on a method call would take me to that method's code... I only had two small source files with changes and it was very frustrating. It was also slow to populate the navigation window while switching between the tabs for those two source files in the editor.

A couple times trying to navigate to a method brought me to the top of the source file instead of to where the method was. In general the IDE simply wasn't responding as fast as I needed it too, and is some cases it was just wrong.

I'm not sure what sort of information is cached by this process, but I find things like getting the call hierarchy to be very slow, even though this scanning is complete. Building a complete database of where every variable and method is referenced should be possible to do faster than compiling the project (at the very least, no slower). Referencing such a database should make things like getting the call hierarchy instantaneous as far as operator perception is concerned.

Posted by swpalmer on July 23, 2009 at 08:56 AM PDT #

> "A couple times trying to navigate to a method brought me to the top of the
> source file instead of to where the method was.

This is the only part of the comment where I can make a competent comment right now (I woke up 5 minutes ago) - this issue is already reported and it is "started":

http://www.netbeans.org/issues/show_bug.cgi?id=164252

Posted by guest on July 23, 2009 at 06:31 PM PDT #

There are lots of reports in bugzilla now with stack traces, and the dev team appears to be working on many of them.

For anyone who sees this happening, you might find that using a non-freeform project will really change your experience. If you create a non-freeform project (put the project data tree off to the side somewhere so you can try this), and then modify the build.xml to have targets in it that call out to your ant/maven-builds where needed, you might find that netbeans is a lot more usable, while they work through these issues.

I've found a lot less problems in my use of netbeans after making this change to 13 some projects that are the core of what I am working on right now.

Freeform is really not a good thing right now because it doesn't consider all the things that we need it to in how the scanning is triggered and how it reacts to changes in libarary jar files.

Non-freeform projects seem to be tracking a lot more things incrementally so that there aren't near as many opportunities for netbeans to get lots about a change in something like a library jar, and decide it has to rescan the world.

Posted by Gregg Wonderly on August 07, 2009 at 06:46 AM PDT #

Free-form projects may have more problems, but as far as the excessive scanning goes they are a red herring. I don't have a single free-form project and yet during my editing sessions the scanning issue comes up constantly.

Posted by Swpalmer on August 09, 2009 at 12:34 PM PDT #

I don't think that freeform projects cause all the problems. But, there are dependencies, which seemed to be tracked better without freeform projects and this can eliminate some of the scanning in my experience.

There are still bugs and architectural issues that need to be addressed to remove all scanning, or at least hide it from view of the editor and other interactive modules that impact the developers work flow.

Posted by Gregg Wonderly on August 10, 2009 at 07:24 AM PDT #

Last comment here was August. It's now half way through October and still no signs that these issues have been fixed, if anything each new release seems worse as, instead of sorting out Netbeans core functionality, new bells and whistles are added to make for a more impressive features list.

I tried scan on demand, but that seems to have increased the "lost classes" problem where the error bagding throws up dozens of errors because the syntax scan has lost track of classes or whole library jars. Also there's no real documentation I can find for scan on demand. When are the libraries excluded by scanondemand.properties actually excluded? Should I exclude the stable libraries I import from a shared drive?

There seems little point in adding more bug reports. Half the remaining Netbeans user community are clearly getting the same things. Nothing has happened to me that hasn't been reported multiple times. And yet the priority seems to remain at P2,

If it's not fixed fairly soon I'll reluctantly have to join the exedous to Eclipse. It's just wasting too much of my time.

Posted by Malcolm McMahon on October 16, 2009 at 06:14 PM PDT #

In NB 6.7.1 my project can be loaded and the IDE is idle for half an hour and as soon as I actually try to something (ctrl-click an identifier for example) - up pops the "Please Wait. Scanning..." dialog. WTF?

Posted by Scott Palmer on October 16, 2009 at 09:09 PM PDT #

Hi everyone,

sorry for not getting to this post for a longer time. I am sorry to hear that many of you are still facing issues with scanning and indexing. If you watch mailing lists, you might have noticed that some of the users do not suffer from their performance problem in 6.7 and some issues were resolved in 6.8 too. I am sorry for maybe too diplomatic reaction but I have to say that the issues are being resolved continually.

This is a list of performance issues that are opened in the relevant categories (hopefully, I didn't forger any):

http://www.netbeans.org/issues/buglist.cgi?Submit+query=Submit+query&issue_type=DEFECT&issue_type=TASK&component=editor&component=java&subcomponent=compiler&subcomponent=editor&subcomponent=Parsing+%26+Indexing&subcomponent=project&subcomponent=source&issue_status=NEW&issue_status=STARTED&issue_status=REOPENED&email1=&emailtype1=exact&emailassigned_to1=1&email2=&emailtype2=exact&emailreporter2=1&issueidtype=include&issue_id=&changedin=&votes=&chfieldfrom=&chfieldto=Now&chfieldvalue=&short_desc=&short_desc_type=substring&long_desc=&long_desc_type=substring&issue_file_loc=&issue_file_loc_type=substring&status_whiteboard=&status_whiteboard_type=substring&keywords=PERFORMANCE&keywords_type=anytokens&field0-0-0=noop&type0-0-0=noop&value0-0-0=&cmdtype=doit&namedcmd=Defects%3A+Editor+Technologies&newqueryname=&order=Reuse+same+sort+as+last+time

I subjectively think this issue is the most important one:

Make IDE usable during scanning - http://www.netbeans.org/issues/show_bug.cgi?id=133943

Also, I would like to overcome presumption that the only thing that we do is that we make IDE more "shiny", disregarding bugs. See the "New and noteworthy" document for 6.8:

http://wiki.netbeans.org/NewAndNoteworthyNB68

You will find that there are mainly improvements from the J2EE/Web area (I don't think the best strategy for us is to ignore J2EE6 or JavaFX) and some improvements by the CND team (C/C++). We really are mainly fixing issues in 6.8...

Posted by Petr Dvorak on October 16, 2009 at 11:43 PM PDT #

Sorry if I got a bit into rant mode there, but this really is getting to be a show-stopper for me. And it's not just one project, it's all of our large, multi-source tree projects.

And it's not just that it locks out Netbeans activities, the intensive IO seems to slow everything on the machine, and it takes ages. It's as if it's not even checking last modified times before completely re-analysing everything in sight.

It might be that the anti-virus software which is on our machines as corporate policy is exacerbating the problem.

Personally I'm not wild about making Netbeans into an all-singing-all-dancing behemoth. It's in danger of entering the "bloatware" category. People aren't very often going to be doing, say, Java and PHP at the same time and, to my mind, it would be cleaner to use different IDEs, but my real concern it that the core functionality seems so badly broken that it ought to be an "all hands to the pumps", priority zero issue, and anyone in the Netbeans team who can contribute should be concentrating on that.

My impression is that you've painted yourself into a corner. You may need a more fundamental rethink of this scanning and indexing stuff than you've been prepared to contemplate. None of us likes to throw code away, but sometimes it's the only way forward, Perhaps what you need to do it to really scan on demand, i.e. check the up-to-date status of a class at the point the information is required. Or how about detecting a period of user inactivity to do some scanning. At the least you need to push down the priority. Maybe take a little rest after each file to allow some other IO to get a look in.

And please give us a control panel for it.

Posted by Malcolm McMahon on October 17, 2009 at 06:16 PM PDT #

Just wanted to say that 6.8 is still VERY slow.

my computer is a Vostro 400 with 4Gb ram and ubuntu 9.10 (with 9.04 or 8.04 is not any better)

the code is about 500k lines of c++ code, admittedly with a lot of templates and classes.

i use boost-build for making the project

i like very much the Netbeans ... but is too slow. Can i turn off some features of the scanning?

Posted by Riccardo Rossi on January 15, 2010 at 05:24 AM PST #

I have recently been trying Netbeans 6.8 on Linux AMD64, with several moderately large projects (manually configured, with makefiles for clearcase clearmake). The C++ scanner is now completely broken. It repeatedly hangs when parsing my source tree, never making any further progress. Sometimes it will complete one project, but it always hangs on something. The GUI remains responsive, and I can still code, but anything that depends on parsing is not usable.
For me this is a major showstopper, and I'd say that Netbeans 6.8 for C++ really ought to be pulled until the parser can be fixed. It is not a usable product, as is.
For now, I'm going back to 6.7. While it is dog slow, at least it works.
Perhaps it is time to stop adding new features, and fix some of the bugs.
Netbeans has some great functionality - if only it were stable, and fast enough.
My machine is a quad core monster, with 8Gb of RAM, and netbeans is still slow. Something is fundamentally wrong.

Posted by Andrew Smith on January 16, 2010 at 04:01 AM PST #

I have just started using NB 6.8 for some C/C++ projects and find that it is working quite well. The projects are not likely as large as yours, but NB 6.8 is already better and faster than MS Visual Studio for C++ development. (If only it supported a compiler that could be used to compile with the Windows SDK, sigh.)

I assume you have reported this bug? The main problem I have with these performance bugs is that it is hard to provide a reproducible test case.. I work on closed-source commercial code, I can't share my projects and I obviously don't have time to create a huge throw-away project just in the hope that it will trigger the performance problems that we see in our "real" work.

6.8 does clearly have performance improvements over 6.7.. but as you say there are tons of bugs still.. I so wish the quality could be improved instead of adding features.

Posted by Scott Palmer on January 16, 2010 at 04:21 AM PST #

My main problem is that since moving out of 6.5 Netbeans marks all my projects with errors... I can reproduce this consistently by opening a project that includes references to an alternate implementation of our API... This essentially makes NetBeans completely useless to us!
It seems that since 6.0 the only thing that changed in NetBeans is support for other languages (such as the FX nonsense) and new features such as the Parsing API were published half baked.
I'll probably move back to 6.5 since its impossible to work like this and having spent some time with 6.8 I see no noticeable improvement...

Posted by Shai on February 10, 2010 at 01:44 PM PST #

I continue to see occasionally long delays in 6.8 when I first uncover the netbeans window after some idle time, or come back to my computer after a hibernated session is restarted. I see the disk activity light on my windows vista laptop just stay lit for a large amount of time, and the whole computer is seemingly unable to do anything sometimes.

I think that this type of behavior is about lack of RAM (2GB on 32-bit OS). Some people may be seeing netbeans behaving in this way due to scanning and parsing and lucene use causing a huge expansion in RSS so that other things are paged out while netbeans pages more things in.

One of the problems with Java is that compiled byte codes are code segments without a disk resident representation. Thus either the JVM has to recompile them on a page in (it does not) or more swap space is used, AND, anytime they need to be paged out, they can't just be dropped as other text can. Instead, they have to be written back to swap space I guess.

At any rate, this issue is really a big deal for a lot of people, and I think that the software is a part of the problem, but there is also OS issues and hardware issues that can complicate really "fixing" this for everyone, without some real data and understanding of where the OS specific issues are and where the Java issues are, and when the netbeans software architecture is actually the problem due to just doing too much stuff in the EDT and otherwise not decoupling long running task away from the users work path.

Posted by Gregg Wonderly on February 16, 2010 at 04:44 AM PST #

SOLUTION! (expensive)

Without SSD drive it takes so long that netbeans crashes! (20 min)
With SSD it takes about 10 seconds!

Get an SSD Drive.

Posted by Keyvan on May 31, 2010 at 02:30 AM PDT #

Post a Comment:
Comments are closed for this entry.
Search

Archives
« April 2014
SunMonTueWedThuFriSat
  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
   
       
Today