Literate programming and cross-reference of formatted tables

[Jacques Shebehe]

Dear members,

Has anyone solved the issue regarding `cross-reference` of formatted tables (html) while using MarkStat `.stmd` or Quarto nbstata `.qmd`.

My workflow is mainly for papers: Stata 17 SE (Mac & Windows) & German Rodriguez MarkStat. Coding/analysis with Mattias Nordin Sublime Text StataEditor package, report rendering to html with MarkStat embedding formatted tables with `.include ` function with great results. Finally, I use html2docx stata command to convert the html to docx.

Unfortunately, cross-reference of tables does not work in MarkStat. ??

As MarkStat uses Pandoc, I tried using pandoc's fenced divs ( :::{#tbl:tableid} ), but it seems that MarkStat does not recognise pandocs fenced divs.


I have recently tried Tim Huegerich nbstata with Quarto, and followed FernandoRios tips in another thread
```{stata}
*| output: asis
display "```{=html}"
type my_table.html
display "```"
```
but this works only if there is no code chunck option for create cross-referencing. Once you specify *| label: tbl-tableid, the document fails to render.

I even tried using the table as `.md` like
::: {#tbl:tableid}
{{< include my_table.md >}}
:::
and this works for simple tables but complex tables are rendered as a chuck of text.


Has anyone found a working solution to cross-reference formatted tables with either MarkStat or nbstata? Have you been able to use Pandoc's cross-referenced fenced divs MarkStat?

Best,

J

[Tim Huegerich]

Quarto can be a little finicky, but there's usually a way to make it do what you want. I'm not able to replicate the "fails to render" with the label added to the code block (without you providing more information), so I'm not sure if this will solve your problem. But try disabling Quarto's html table processing: https://quarto.org/docs/authoring/ta...ble-processing

Here's some sample code that works for me, including a table label.

I use a helper program, defined in a separate code chunk:

```{stata}
*|include: false
capture prog drop qtfix
prog qtfix
syntax anything [, *]
disp "```{=html}"
disp `"<table border="0" width="100%" data-quarto-disable-processing="true">"' 
disp "<tr><td colspan=20><hr></td></tr>"
`anything', `options'
disp "<tr><td colspan=20><hr></td></tr>"
disp "<tr><td colspan=20>"
disp "Standard errors in parentheses"
disp "</td></tr>"
disp "</table>"
disp "```"
end
```

Then here's the actual table code block, which uses esttab to output the html table:

```{stata}
*| label: tbl-treat
*| output: asis
qui {
* [some setup code here omitted]
}
qtfix esttab m2lte m3lte m2 m3 m2non m3non, mtitles keep(treat_effect treat_period) b(%7.3f) se(%7.3f) width(100%) align(center) html fragment nostar compress
```

[FernandoRios]

Hi Jacques
So, I guess the biggest point here is if you are using Stata dynamically (so it generates the table) or if you are using stata to create the code, then do something else with the "saved" text.
seems you are trying to do the latter. Use Stata to create the tables, then add them to your document.

In that case, i find the following code the simplest

:::{#tbl-tabletag1}

{{< include table-md>}}

note to table (unless its already in the table)

title for table
:::

for HTML is a bit different:

:::{#tbl-tabletag2}

```{=html}
{{< include table-html>}}
```
note to table (unless its already in the table)

title for table

:::

or just using Latex

:::{#tbl-tabletag3}


{{< include table-tex>}}

note to table (unless its already in the table)

title for table

:::

Notice that i do not add an extension, since you don't really need it.

Then, simply @tbl-tabletag1 or @tbl-tabletag2 or @tbl-tabletag3 should do the trick.

If interested, i have a set of slides on this here https://riosavila.github.io/rm-data/...01/week01.html

[Jacques Shebehe]

Thank you, Dear Tim!

I was able to produce the table with esttab command and the helper program.

Is it possible to implement the approach for html tables produced using the collect suite of commands? I tried this `qtfix collect export tab4, as(html) replace` as below but it didn't work.

qui webuse auto, clear

A separate code chunk for the helper program

```{stata}
*| label: tbl-qtfix
*| tbl-cap: "Html table as per helper program"
*| output: asis
*| echo: false

qui table foreign, stat(mean price) stat(sd price)
qui collect layout (foreign) (result)
qui collect style cell cell_type[row-header], font(, bold)
qtfix collect export tab4, as(html) replace       
```

[Jacques Shebehe]

Hi Fernando,
Many thanks!

In my workflow, I use Stata to create and test codes weaving codes and narrative in Sublime Text, and then produce report from the Stata commad line (using MarkStat) and recently from Terminal in Sublime Text/VS Code (using nbstata and Quarto).

The approach you shared worked perfectly first after I had the tables (html and md) already saved in the current directory. It was also possible to {{< include >}} files located in a subfolder of the current directory. I also need to specify the extension (.html and .md) `{{< include file_name.html>}}` for the qmd to render.

Consequently trying to render a quarto manuscript project did not work: I get an error that the files do not exist. Any ideas about other alternatives in case of quarto manuscript/book project and table rendering?

[Jacques Shebehe]

Hi Tim,

One more question! I tried to replicate the approach even for rendering a quarto manuscript project, but it failed. Below is the content of the `index.qmd` file.

---
title: HTML tables
authors:
  - name: Norah Jones
    affiliation: The University
    roles: writing
    corresponding: true
bibliography: references.bib
jupyter: nbstata
---

## Section
This is a simple placeholder for the manuscript's main document [@knuth84].

```{stata}

qui webuse auto, clear

```

```{stata}
*| include: false

capture prog drop qtfix
prog qtfix
syntax anything [, *]
disp "```{=html}"
disp `"<table border="0" width="100%" data-quarto-disable-processing="true">"' 
disp "<tr><td colspan=20><hr></td></tr>"
`anything', `options'
disp "<tr><td colspan=20><hr></td></tr>"
disp "<tr><td colspan=20>"
disp "Standard errors in parentheses"
disp "</td></tr>"
disp "</table>"
disp "```"
end
```

@tbl-qtfix produced as per helper program.

```{stata}
*| label: tbl-qtfix
*| tbl-cap: "Html table as per  helper program"
*| output: asis
*| echo: false

eststo: quietly regress price weight mpg
eststo: quietly regress price weight mpg foreign
qtfix esttab, ar2 align(center) html fragment nostar compress
```

[Tim Huegerich]

This is working for me, except that I can't get Table 1 centered...

```{stata}
*| echo: false
set linesize 255
qui sysuse auto, clear

qui table foreign, stat(mean price) stat(sd price)
qui collect layout (foreign) (result)
qui collect style cell cell_type[row-header], font(, bold)
qui collect export tab4, as(html) replace
```

:::{#tbl-first}

{{< include tab4.html >}}

Html table
:::

@tbl-first produced without unneeded helper program.

```{stata}
*| label: tbl-second
*| tbl-cap: "Another Html table"
*| output: asis
*| echo: false
capture prog drop qtfix
prog qtfix
syntax anything [, *]
disp "```{=html}"
disp `"<table border="0" width="100%" data-quarto-disable-processing="true">"'
disp "<tr><td colspan=20><hr></td></tr>"
`anything', `options'
disp "<tr><td colspan=20><hr></td></tr>"
disp "<tr><td colspan=20>"
disp "Standard errors in parentheses"
disp "</td></tr>"
disp "</table>"
disp "```"
end

qui eststo: quietly regress price weight mpg
qui eststo: quietly regress price weight mpg foreign
qtfix esttab, ar2 align(center) html fragment nostar compress
```

[Jacques Shebehe]

For me, it does not work as expected. 2 observations rendering this last code in post #7

It seems that successfully producing a formatted cross-referenced table in html depends on the Stata command you use to produce the table and whether you are rendering a qmd document or a project (I've so far tested manuscript project).

I. collect suite vs esttab

1. tables produced by `collect command`: When {{< include file >}} is in the file, Quarto seems to start by searching for the file and stops if the file is not present in the working directory without even starting nbstata Kernel. This what happened when render the qmd file as per post #7 with the expectation that quarto should produce tab4.html and embedd it dynamically.

But I got this error:

ERROR: Include directive failed.
in file auto3.qmd,
could not find file /Users/path_to_working_directory/tab7.html.
Stack trace:
in file auto3.qmd,
could not find file /Users/path_to_working_directory/tab7.html.
at retrieveInclude (file:///Applications/quarto/bin/quarto.js:39440:19)
at standaloneInclude (file:///Applications/quarto/bin/quarto.js:39476:11)
at processMarkdownIncludes (file:///Applications/quarto/bin/quarto.js:39706:38)
at expandIncludes (file:///Applications/quarto/bin/quarto.js:39735:11)
at async projectResolveFullMarkdownForFile (file:///Applications/quarto/bin/quarto.js:40370:25)
at async fileExecutionEngine (file:///Applications/quarto/bin/quarto.js:41840:26)
at async fileExecutionEngineAndTarget (file:///Applications/quarto/bin/quarto.js:41858:20)
at async render (file:///Applications/quarto/bin/quarto.js:82934:24)
at async Command.actionHandler (file:///Applications/quarto/bin/quarto.js:83077:32)
at async Command.execute (file:///Applications/quarto/bin/quarto.js:8017:13)

2. Regarding tables produced by esttab (helper program is needed, see post #3 ): no need to have the tables already saved, they can be produced and rendered/embedded dynamically in single qmd file (but not in a project, e.g. manuscript project).

II. single qmd vs (manuscript) project.

1. With {{< include file.html >}}, tables that are already in the project directory are embedded correctly in both signle document and project. See post #3

2. esttab and helper program seems not to work in Quarto (manuscript) project.

Rendering this manuscript project `index.qmd`

---
title: HTML tables
authors:
- name: Norah Jones
affiliation: The University
roles: writing
corresponding: true
jupyter: nbstata
---
## Section
@tbl-second produced with helper program.
```{stata}
*| label: tbl-second
*| tbl-cap: "Another Html table"
*| output: asis
*| echo: false
capture prog drop qtfix
prog qtfix
syntax anything [, *]
disp "```{=html}"
disp `"<table border="0" width="100%" data-quarto-disable-processing="true">"'
disp "<tr><td colspan=20><hr></td></tr>"
`anything', `options'
disp "<tr><td colspan=20><hr></td></tr>"
disp "<tr><td colspan=20>"
disp "Standard errors in parentheses"
disp "</td></tr>"
disp "</table>"
disp "```"
end
qui webuse auto, clear
qui eststo: quietly regress price weight mpg
qui eststo: quietly regress price weight mpg foreign
qtfix esttab, ar2 align(center) html fragment nostar compress
```

I got this error

Executing 'index.quarto_ipynb'
  Cell 1/1: 'tbl-second'...Done
Rendering qmd embeds [index.qmd]
Rendering output notebook [index.qmd]
Error running filter /Applications/quarto/share/filters/main.lua:
Block, list of Blocks, or compatible element expected, got table
        while retrieving function argument content
        while retrieving arguments for function Div
stack traceback:
        /Applications/quarto/share/filters/main.lua:22368: in field 'render'
        /Applications/quarto/share/filters/main.lua:1233: in local 'filter_fn'
        /Applications/quarto/share/filters/main.lua:635: in function </Applications/quarto/share/filters/main.lua:625>
        (...tail calls...)
        [C]: in ?
        [C]: in method 'walk'
        /Applications/quarto/share/filters/main.lua:557: in function </Applications/quarto/share/filters/main.lua:546>
        (...tail calls...)
        /Applications/quarto/share/filters/main.lua:1334: in local 'callback'
        /Applications/quarto/share/filters/main.lua:1352: in upvalue 'run_emulated_filter_chain'
        /Applications/quarto/share/filters/main.lua:1388: in function </Applications/quarto/share/filters/main.lua:1385>
stack traceback:
        /Applications/quarto/share/filters/main.lua:557: in function </Applications/quarto/share/filters/main.lua:546>
        (...tail calls...)
        /Applications/quarto/share/filters/main.lua:1334: in local 'callback'
        /Applications/quarto/share/filters/main.lua:1352: in upvalue 'run_emulated_filter_chain'
        /Applications/quarto/share/filters/main.lua:1388: in function </Applications/quarto/share/filters/main.lua:1385>

Please, correct me if I am wrong or if you have other tricks!

I am very grateful to you dear Tim Huegerich and FernandoRios. I have a least one working solution (post #2 and #3) even if I will need to change my workflow a bit.

[FernandoRios]

this works for me.
basically add lines between text and fences and Hashtags

---
title: HTML tables
authors:
  - name: Norah Jones
    affiliation: The University
    roles: writing
    corresponding: true
jupyter: nbstata
---

## Section

This is a simple placeholder for the manuscript's main document .

```{stata}
qui webuse auto, clear
qui table foreign, stat(mean price) stat(sd price)
qui collect layout (foreign) (result)
qui collect style cell cell_type[row-header], font(, bold)
qui collect export tab2.html, replace
qui table foreign, stat(mean price) stat(sd price)
qui collect layout (foreign) (result)
qui collect style cell cell_type[column-header], font(, bold)
qui collect export tab1.md, replace
```

While @tbl-tabletag1 refers to md table,

::: {#tbl-tabletag1}

```{stata}
*| echo: off
*| output: asis
type tab1.md
```
note to table (unless its already in the table)
Caption of  md table
:::

@tbl-tabletag2 refers to html table.

::: {#tbl-tabletag2}
```{stata}
*| echo: off
*| output: asis
display "```{=html}"
type tab2.html
display "```"
```
note to table (unless its already in the table)
Caption of html table
:::

[Tim Huegerich]

I don't think there is any reason something shouldn't work in a manuscript/project that works in a single qmd. I suspect it's something specific in the configuration of your manuscript project that is causing the difference, such as the freeze option: https://quarto.org/docs/manuscripts/...-code-executed

In any case, these are now Quarto issues you are encountering, not Stata (or nbstata) issues. So you may get better help asking at Quarto Discussions: https://github.com/quarto-dev/quarto-cli/discussions

[Jacques Shebehe]

Thanks a lot for all help. I see Quarto is capricious. I will carefully look into the configuration of my project and if need be ask in a Quarto Group.