{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/natural-language-generation-with-neural","title":"Natural Language Generation with Neural Variational Models","arxiv_id":"1808.09012","date":"2018-08-27","proceeding":null,"authors":["Hareesh Bahuleyan"],"abstract":"In this thesis, we explore the use of deep neural networks for generation of\nnatural language. Specifically, we implement two sequence-to-sequence neural\nvariational models - variational autoencoders (VAE) and variational\nencoder-decoders (VED). VAEs for text generation are difficult to train due to\nissues associated with the Kullback-Leibler (KL) divergence term of the loss\nfunction vanishing to zero. We successfully train VAEs by implementing\noptimization heuristics such as KL weight annealing and word dropout. We also\ndemonstrate the effectiveness of this continuous latent space through\nexperiments such as random sampling, linear interpolation and sampling from the\nneighborhood of the input. We argue that if VAEs are not designed\nappropriately, it may lead to bypassing connections which results in the latent\nspace being ignored during training. We show experimentally with the example of\ndecoder hidden state initialization that such bypassing connections degrade the\nVAE into a deterministic model, thereby reducing the diversity of generated\nsentences. We discover that the traditional attention mechanism used in\nsequence-to-sequence VED models serves as a bypassing connection, thereby\ndeteriorating the model's latent space. In order to circumvent this issue, we\npropose the variational attention mechanism where the attention context vector\nis modeled as a random variable that can be sampled from a distribution. We\nshow empirically using automatic evaluation metrics, namely entropy and\ndistinct measures, that our variational attention model generates more diverse\noutput sentences than the deterministic attention model. A qualitative analysis\nwith human evaluation study proves that our model simultaneously produces\nsentences that are of high quality and equally fluent as the ones generated by\nthe deterministic attention counterpart.","url_abs":"http://arxiv.org/abs/1808.09012v1","url_pdf":"http://arxiv.org/pdf/1808.09012v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"natural-language-generation-with-neural","repo_url":"https://github.com/HareeshBahuleyan/tf-var-attention","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"tf","reach":null}],"tasks":[{"task_slug":"text-generation","task_name":"Text Generation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1808.09012","atlas_url":"https://app.syntology.ai/?focus=1808.09012","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"1808.09012"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"1808.09012\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"1808.09012"},"call":"get_concepts_for_paper(arxiv_id=\"1808.09012\")"}],"by_repo":[{"repo":"HareeshBahuleyan/tf-var-attention","kind":"official","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["1712.08207"],"harvested_for_calls":[{"arxiv_id":"1712.08207","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"1712.08207"},"call":"get_harvested_code_for_paper(arxiv_id=\"1712.08207\")"}]}],"by_repo_harvested_for_is":"harvested_for: the arXiv ids (as Syntology's graph holds them) of every paper whose samples Syntology's graph links to this repository, in arXiv-id order, complete; the page names the first 3 and says how many more. A link only: it does not say the papers are the same work, that one copied the other, or that the code implements this paper's method; harvested_for_calls gives the get_harvested_code_for_paper call for each arXiv-shaped id among them","by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"0140cd2362084209db55f5c6d110306b33fb532be09e3d20e58937e2df9f1750","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}