The "Progress" panel draws what a statement is doing while it does it: every step of the execution as a card, the rows flowing between them as lines, and what each step has cost in time, calls and bytes on the card itself. It replaces the "Explain Plan" panel, which was meant to show how a statement was going to be executed and in practice showed nothing at all. The panel answers the question a statement of several minutes raises: what is it busy with. Where the [[Invantive Query Tool/Trace|Trace]] panel says what happened, one message per row, and [[Invantive Query Tool/History|History]] says what a statement cost once it is over, this one is the picture while it runs. ![[20260905-qt-progress-panel.png]] ## Switching It On Collecting is off when the application starts. The button "Progress" switches it on and off, the way the button "Trace" does in the panel beside it, and the lamp on the button is green while it is on. A statement executed while it is off is not drawn, and nothing is kept: the panel then costs no memory, no reading and no time in the statement. Switch it on before executing the statement to be watched. The panel belongs to the document being edited. Each document draws its own statement, and switching to another document shows that document's picture as it was left. ## What a Card Says One card per step of the statement. A step is either an operator - a join, a filter, an ordering, a grouping - or the reading of one table of a data container, however many partitions that reading covers. A card headed "Result set" is neither: it is a node the plan puts around the rows of a branch, and it reads nothing and makes no call. It hands on what the step beneath it produced, and it is drawn because the rows and the time on it are measured separately from the step it wraps. The outcome of a statement inside a block or a loop, such as the number of rows an insert wrote, is a card of this kind as well, with the write and the query it took its rows from beneath it. - The glyph on the disc says which operator it is. The disc is ringed in a colour which says how busy the step is, using the same five bands as the [[Invantive Query Tool/Connection Monitor|Connection Monitor]]. - A step which reads or writes a table says which data container it does it on and which object: `eol - TRANSACTIONLINESINCREMENTAL`. Where the statement gave the table an alias of its own it follows in brackets, since that is what tells two reads of one table apart. - The shape in the top right corner says what the step is doing: a clock face waiting for the step before it, a triangle running, a tick done, a cross failed. It says the same as the colour of the ring, so either one is enough. - Under the name stand the rows the step has handed on and how long it has been at it, and under those the calls it has made and the bytes it moved: what a read received, and what a write sent, since a write sends its rows and receives an acknowledgement of little more than nothing. The export as a report keeps the two directions apart in two columns. A duration of less than ten seconds is written in milliseconds, since a statement answered out of a cache is done in a few of them and tenths of a second would show every step as having taken no time at all. - The strip along the bottom says where the time went: computing, waiting on a data container, waiting on a rate limit, waiting because the platform refused, waiting for room in a buffer, and waiting because the user is holding the step back. - A step which covers several partitions carries a band saying how many are done, how many are running and how many are still queued. - A step with a buffer between what reads and what consumes carries a gauge saying how full it is, whether it reads one partition or a thousand. This is what a read which has taken in fifty megabytes and handed on nothing yet is doing: the rows are in the buffer, waiting for whatever consumes them to ask for the next one. - A step which failed is drawn in red, with the message in its tooltip. The tooltip of a card carries the whole story of that step in text, which is also what the export writes. A long text, such as the statement itself, is shortened there to twenty lines, its beginning and its end; the whole of it is on the tab "SQL" of the window "Details". A step which has been held back, paced, stopped after a number of rows or peeked into says so on its card, in a chip beside the gauge: an instrument which changes what a statement does and leaves the picture unchanged would be one nobody could trust. The help of the panel - the question mark at the right of the tool strip - opens with the numbered legend of both kinds of card: fourteen marks on the card of a step and eleven on the card of a platform, each explained. It is the page to read first. ## A Read over Files Available from release 28.0. A driver which reads files, such as Big Data JSON, reads every file on its own, several side by side. Its card counts files where other reads count partitions, and the band says so: `21,390 of 42,625 files · 32 active`, the files still queued being the grey part of the band. A partition of such a driver is a folder, and "Show Partitions" lists the folders. Under the band stand up to three lines, each only when it has something to say: - The bytes read of the bytes on disk to read, an estimate of the time left, and the average rate since the reading began: `4.1 GB of 9.4 GB · about 10 min left · 16.2 MB/s`. The time left is the bytes still to read at the rate so far, with the files being read counted as half read. It appears after ten seconds of reading and one file read to the end, and not while a step is held back. Where the line is longer than the card, the rate is left out; the tooltip always gives it. The bytes of a file count when the file has been read, so one large file shows its bytes at its end. - What the index of the files did. "Index: 267,292 skipped" counts the files the index proved to hold no matching row. "Unindexed" counts the files the index does not describe as they are now. These files are read in full, whatever the filters, and "described" counts the ones this read added to the index. "Index: no filter to skip files on" means that no filter of the statement can rule out a file. "Index: off" means that the use of the index is switched off. - The files which were gone since the folder was listed and were skipped, and the files which could not be read and were skipped as `max-erroneous-files` allows. The file the statement failed on is named in red. Before the first file is opened, the card says what happens instead: "Listing files" and then "Choosing from 306,816 files", each with the time spent so far. A large folder takes seconds to list, and the index is consulted before the reading starts. The rows on such a card read `0 of 150,123,456 rows · 7:59`: the rows the step handed on, of the rows the records of its files gave before any filter. A read of millions of records which hands on nothing is busy, not stuck, and the two figures together show it. The tooltip repeats all of this, with the bytes the index skipped and the time the listing and the choice took. ## The Platform Behind the reads of a data container stands a card for the platform itself, with the connector it speaks to drawn large. It carries the calls made to that platform and the bytes moved, and under those a chip for every rule the platform holds the statement to: what the rule allows, over which period, and whether it counts per partition or over the whole connection. A chip turns amber as its allowance fills and red once it has actually made the statement wait. A rule which counts per partition is one rule however many partitions it applies to, so it is one chip: it says how many partitions the rule is in use for and how full the fullest of them is. A statement over a thousand divisions therefore adds one line to the card and not a thousand. Calls the platform refused are counted in red on the card. A platform which is the reason the statement is not going faster is marked as limiting, rather than the read which is waiting for it, since it is the platform which decides what can be changed. The figures of these chips are the same figures `systemdatacontainerslotbasedratelimiters@datadictionary` answers with, read from the same place. ## Moving About - The picture stands in the middle of the panel, with the same room on the left as on the right and above as below, and it stays there while the statement grows it. A picture larger than the panel starts a little in from the edge it is read from and the rest is scrolled to. - Panning, zooming, "Fit" and reaching a card with the keyboard all hand the view over: from then on the picture stays where it was put. The next statement, "Re-layout", "Wrap" and "Clear" place it in the middle again. - The wheel of the mouse scrolls the picture up and down, with Shift held left and right, and a wheel which tilts scrolls sideways of its own accord. Held with Control, the wheel zooms. A picture which fits the panel does not scroll. - Cards may be dragged. "Re-layout" puts every card back where it belongs. - "Fit" zooms until the whole picture is in view; the two buttons beside it zoom in and out. - "Wrap" lets the chain turn aside at the edge of the panel and read back on the next row, instead of scrolling. Off, the chain stays on one row. - "Rotate" turns the picture a quarter clockwise. The statement moves from the left to the top, to the right, to the bottom and back to the left, and the columns of the picture follow it, so four presses come back where they started. Where the statement stands at this moment is in the tooltip of the button and is what a screen reader says of it. A screen standing on its side, and a panel docked narrow and tall, have room for a chain running down them which they do not have for one running across; the cards themselves stay upright, whichever way the picture stands. "Wrap" works in every orientation and measures itself against the side the picture grows along. - Scroll bars appear along the bottom and the right as soon as the picture is larger than the panel, so how much of the statement lies off the edge can be seen and reached without dragging. - "Export" writes what is shown to a file in one of five ways: as text, the tree indented one step per level; as JSON, the whole reading; as a picture, the graph as it stands in one self-contained file; as a report, that picture together with the tables of figures in one HTML file; or as a Mermaid diagram. - The Mermaid diagram is the one export which describes the picture instead of drawing it: a few lines of text which a wiki page, a ticket, a pull request, a page of documentation and every editor which knows Mermaid draw for themselves, and which can be edited afterwards - a branch dropped, a step renamed, the three cards which matter kept. It carries the cards with their figures, the lines with the rows and the bytes on them, the platforms with their rules, and the state of every step as a colour. Where a card was dragged to does not travel with it, since Mermaid places the nodes itself; the orientation does, so a picture turned to run down the screen is written as a diagram which runs down the page. ## Steps Which Do the Same Thing A statement can make thousands of steps: a loop makes a branch per iteration and a parallel construct one per worker. Steps which stand under the same step and do the same thing with the same parameters are therefore drawn as one card from nine of them on, with a badge saying how many. This happens by itself and is not something to switch on. The card carries the figures of all of them together. Its duration is the time during which at least one of them was running: their sum when they ran one after another, as the iterations of a loop do, and less when they ran side by side. Pressing the badge - or "Show the Identical Steps" in the menu of the card - lists them one row each, with the rows, the duration, the calls and the failure of each, which answers which of them was slow and which of them failed. Of a run of more than twenty such steps the list shows the first ten and the last ten, those still running and at most ten which failed, each numbered by its place in the run; the card and the status line count every one of them. Beneath the card, what those steps did is drawn as one branch in the same way. The read which every iteration of a loop made is one card with a badge saying how many iterations made it, also when that is fewer than nine, and a step which only one iteration made is drawn on its own. The platforms which those reads and writes talk to stand behind them, as anywhere else in the picture. "Show Calls" on such a card lists the calls of every iteration. ![[20260905-qt-progress-merged.png]] ## Folding Away What Has Finished A long statement ends with a picture whose left-hand side is still moving and whose right-hand side finished long ago. Folding empties the finished side, so that what is left on the screen is what is still happening. - The chevron under the glyph of a card folds that card and everything feeding it into one card, and opens them again. It appears once every step of that branch has finished, since a step which is still running is never hidden. - "Fold Done" folds the deepest layer of finished steps, and on each further press the layer above it, so the picture empties from the right. "Unfold" opens the layer folded last. - A folded branch is one card: how many steps it stands for, which kinds of step those are, and the rows, duration, calls and bytes of the whole branch. The line to the card it feeds carries what the branch delivered, and the data containers it talked to keep their lines to it. - An export writes the picture as it stands, folds included. ![[20260905-qt-progress-folded.png]] ## Holding a Statement Back The panel can also decide when the rows move. This costs the statement nothing when it is not used and changes nothing about what the statement computes; it only decides when rows are handed on and whether a copy of one is kept to look at. - "Pace" sets how fast every step of the next statement may hand on its rows, from full speed down through so many rows a second to frozen, which is the order the list offers them in. The setting holds for the statements which follow until it is changed. - "Freeze All" holds every step. Set before a statement is executed, the statement builds itself and stops: the picture shows what it is going to do without a single call having been made. "Unleash All" lets it go. - "Release 10 Rows" lets every held step hand on ten rows and freeze again. Up to and including release 27.0 this button reads "Step 10", and the same entry in the menu of a card reads "Hand On Ten Rows". The menu of a card opens from the button with the three dots at its top right, and with the right mouse button on the card. It carries the same instruments for one step alone, so one read can be held while the rest of the statement runs, and two more besides: - "Freeze at a Value..." holds the step as soon as a row passes whose named column holds the value given. One wrong row in a read of a hundred thousand rows is found this way, not by releasing ten rows at a time. - "Stop After a Number of Rows..." ends the step once it has handed on that many rows and lets the statement finish with what it has. This is the one instrument which changes what the statement returns, and the card, the status line and the export all say that the statement was cut short. A card of a write carries one more instrument, "Freeze the Writes". A write is an insert, an update, a delete, a synchronisation or the creation of a table from a select. The entry freezes every card of a write in the picture. This is the dry run of a statement that changes data: while the write is frozen, nothing is sent to the data container. The other instruments act on the write itself, so the panel holds, paces, counts or keeps a copy of each row on its way to the target. "Release 10 Rows" lets ten rows through, and "Stop After a Number of Rows..." ends the write after that many rows and lets the statement finish with them. A write without rows is held before its call. An update or a delete which the data container performs from a filter is such a write. A synchronisation is held once, after both sides have been read and compared and before the first change. A write which sends its rows in batches, such as a bulk insert, sends a batch when all rows of that batch have passed. What the select does meanwhile depends on how the write takes its rows. A bulk insert and a delete take their rows from a finished list, so the select runs to its end and the rows wait. An insert and an update take their rows as they arrive, so the select stands still behind the row which is held. The creation of a table does either, depending on the data container. The card of a write exists only after the write has started. To freeze a write from the beginning, set the pace to frozen before the statement runs and choose "Freeze the Writes" on the card which then appears. Then set the pace to full speed, so that the rest of the statement goes on. "Freeze All" holds a write statement before the write is made, so the picture draws no card of the write. "Unleash All" lets the write go as well. The card of a platform hands on no rows of its own, so its menu carries none of these instruments: only "Show Details" and "Show Calls". On the card of identical steps the instruments act on all of those steps together, also on the steps which start later in the same run, such as the read of the next iteration of a loop; "Freeze This Step" reads "Freeze These Steps" there. The rows are counted over all of them, as the card counts them: "Pace" sets how fast they hand on rows together, "Release 10 Rows" lets ten rows through among them, a row which matches "Freeze at a Value..." in any of them holds all of them, and "Stop After a Number of Rows..." ends each of them once they have handed on that number together, so a step which starts after that hands on nothing. "Peek at the Rows" keeps the rows of all of them in one list. The instruments act on a statement which started while the panel was switched on. For any other statement they are greyed out, with that reason in their tooltip. ## Looking at the Rows and the Calls The menu of a card also opens a drawer under the picture, which sorts, filters, groups and searches exactly as the [[Invantive Query Tool/Results|Results]] panel does. Its columns carry the names the rest of the product uses, counts and byte totals are ranged right so they can be compared down a column, and a moment or a short code is ranged in the middle. The bar between the picture and the drawer can be dragged, so a long list can be given most of the panel and a short one very little: - "Show Partitions" lists the partitions the step reads or writes, one row each, with the rows, the calls and the bytes it moved and how fast it moved them. The figures are those of the [[Invantive Query Tool/Connection Monitor|Connection Monitor]] for the same partition. The entry follows the tab "Partitions" of "Show Details" below: it is left out where that tab is, and greyed out with the reason for a step which reads one partition. - "Show Calls" lists the calls the step made to the platform, one row each, with what was sent and received, how long each took and what any of them said when it failed. This is the answer to a card which says a hundred calls where ten were expected. At most a thousand calls are listed: every call still running, every failed call and the newest of the rest. The title says so when the step made more, and the [[Invantive Query Tool/Connection Monitor|Connection Monitor]] keeps all of them. On the card of identical steps the list holds the calls of all of them, also of those the card only counted, on the card of a folded branch the calls of every step of that branch, and on the card of a platform the calls of every step drawn with it, so the list always holds the calls the card counts. "Show Partitions" lists the partitions of the same steps. - "Peek at the Rows" keeps a copy of the rows the step hands on and shows them, under the names the columns carry in the statement, with the alias of the table where two of them share a name. At most a hundred rows are kept, and at most a megabyte, so a statement carrying documents does not fill the memory; at least one row is always kept, however large it is. ![[20260905-qt-progress-drawer.png]] "Show Details" at the top of the same menu opens a window of its own beside the panel, with everything the panel knows of the step. "General" holds what the tooltip of the card says, "SQL" the whole text of the step and of the statement it belongs to, and "Calls" and "Partitions" the same lists as the drawer. The window stays open and follows the statement while it runs. "Partitions" is left out for a step which talks to no platform, such as a join or the statement as a whole, for a step on a platform without partitions, and for the card of a platform, which counts no partitions; a step which reads one partition of a platform which has several has nothing to list there, and that tab says so. ## The Help of the Panel "Help" on the toolbar of the panel opens a window which explains what the panel draws, in the appearance the application is in and in the language it is set to. Its first page carries an anatomy sheet for each of the two kinds of card, and a third for the card of a read over files: the card itself, drawn at twice its size, with a numbered line from every mark on it to what that mark means. Nothing on the sheet is a picture: the card is drawn by the same means the panel draws with, so the sheet says what the panel does rather than what it did when a screenshot was taken. The pages which follow list the glyphs of the operators, the writes, the partitions, the platform, the pace and the drawer, each with what it means. The window closes with the Escape key. ![[20260905-qt-progress-help.png]] ## What the Panel Does Not Do The panel shows what is happening. There is no estimate of what a statement will cost before it runs: the engine estimates nothing, and a figure invented here would be a guess presented as a measurement. An estimate appears only where it follows from what the statement has measured so far, and it says that it is an estimate, such as the time a read over files has left. The nearest thing to a plan is "Freeze All" before executing the statement, which draws the shape without executing any of it. The picture is limited to 500 cards. Steps which do the same thing are counted on their one card however many there are, so a loop of ten thousand iterations or a table function called for every row fits; a tree of more than 500 steps which all differ is drawn up to that point, and the status line says how many were left out.