{"about":{"non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","site":"https://codewithpapers.app","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page","syntology":{"site":"https://syntology.ai","developers":"https://syntology.ai/developers","mcp":{"server":"https://syntology.ai/mcp","transport":"streamable-http","server_card":"https://syntology.ai/.well-known/mcp/server-card.json","auth":{"type":"trial token, no account","trial_token":"https://syntology.ai/api/oauth/trial/token","method":"POST","docs":"https://syntology.ai/developers"}},"have":"https://syntology.ai/api/graph/have?x=<method, arXiv id or title> (free, answers coverage only)","paper_base":"https://syntology.ai/paper/","atlas_base":"https://app.syntology.ai/?focus="},"machine_readable":[{"url":"https://codewithpapers.app/llms.txt","what":"the machine catalog: every machine-readable file, counted"},{"url":"https://codewithpapers.app/index/manifest.json","what":"paper-to-code index by arXiv id, with Syntology's counts"},{"url":"https://codewithpapers.app/search/manifest.json","what":"site search index (titles, authors) and its files"},{"url":"https://codewithpapers.app/download","what":"bulk files: Syntology's layer, described there"},{"url":"https://codewithpapers.app/build_manifest.json","what":"the build record: inputs, counts, exclusions, probes"}]},"url":"/paper/time-contrastive-networks-self-supervised","title":"Time-Contrastive Networks: Self-Supervised Learning from Video","arxiv_id":"1704.06888","date":"2017-04-23","proceeding":null,"authors":["Pierre Sermanet","Corey Lynch","Yevgen Chebotar","Jasmine Hsu","Eric Jang","Stefan Schaal","Sergey Levine"],"abstract":"We propose a self-supervised approach for learning representations and\nrobotic behaviors entirely from unlabeled videos recorded from multiple\nviewpoints, and study how this representation can be used in two robotic\nimitation settings: imitating object interactions from videos of humans, and\nimitating human poses. Imitation of human behavior requires a\nviewpoint-invariant representation that captures the relationships between\nend-effectors (hands or robot grippers) and the environment, object attributes,\nand body pose. We train our representations using a metric learning loss, where\nmultiple simultaneous viewpoints of the same observation are attracted in the\nembedding space, while being repelled from temporal neighbors which are often\nvisually similar but functionally different. In other words, the model\nsimultaneously learns to recognize what is common between different-looking\nimages, and what is different between similar-looking images. This signal\ncauses our model to discover attributes that do not change across viewpoint,\nbut do change across time, while ignoring nuisance variables such as\nocclusions, motion blur, lighting and background. We demonstrate that this\nrepresentation can be used by a robot to directly mimic human poses without an\nexplicit correspondence, and that it can be used as a reward function within a\nreinforcement learning algorithm. While representations are learned from an\nunlabeled collection of task-related videos, robot behaviors such as pouring\nare learned by watching a single 3rd-person demonstration by a human. Reward\nfunctions obtained by following the human demonstrations under the learned\nrepresentation enable efficient reinforcement learning that is practical for\nreal-world robotic systems. Video results, open-source code and dataset are\navailable at https://sermanet.github.io/imitate","url_abs":"http://arxiv.org/abs/1704.06888v3","url_pdf":"http://arxiv.org/pdf/1704.06888v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"time-contrastive-networks-self-supervised","repo_url":"https://github.com/araffin/srl-zoo","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"unanswered"}},{"paper_slug":"time-contrastive-networks-self-supervised","repo_url":"https://github.com/elicassion/3dtrl","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"time-contrastive-networks-self-supervised","repo_url":"https://github.com/sermanet/tcn","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"unanswered"}},{"paper_slug":"time-contrastive-networks-self-supervised","repo_url":"https://github.com/tensorflow/models","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"time-contrastive-networks-self-supervised","repo_url":"https://github.com/tensorflow/models/tree/master/research/tcn","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"time-contrastive-networks-self-supervised","repo_url":"https://github.com/2023-MindSpore-1/ms-code-7/tree/main/TCN","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"mindspore","reach":null},{"paper_slug":"time-contrastive-networks-self-supervised","repo_url":"https://github.com/tensorflow/models/tree/archive/research/tcn","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"tf","reach":null}],"tasks":[{"task_slug":"metric-learning","task_name":"Metric Learning"},{"task_slug":"reinforcement-learning","task_name":"Reinforcement Learning"},{"task_slug":"reinforcement-learning-1","task_name":"Reinforcement Learning (RL)"},{"task_slug":"self-supervised-learning","task_name":"Self-Supervised Learning"},{"task_slug":"video-alignment","task_name":"Video Alignment"},{"task_slug":"reinforcement-learning-2","task_name":"reinforcement-learning"}],"methods":[{"method_slug":"1cycle","method_name":"1cycle"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/video-alignment-on-upenn-action","task":"Video Alignment","dataset":"UPenn Action","model":"TCN","rank_in_archive_order":3,"of":4,"metrics":{"Kendall's Tau":"0.7353"},"uses_additional_data":false}],"syntology":{"syntology_url":"https://syntology.ai/paper/1704.06888","atlas_url":"https://app.syntology.ai/?focus=1704.06888","mcp":null,"developers":"https://syntology.ai/developers","held":true,"reason":"arxiv_id_in_atlas","checked_against":{"atlas_ids":549449,"atlas_ids_sha256":"afd72399dafd9906e9062927f70655952385a3d75488558e1f0484e950ab09b7","read_at":"2026-10-05T01:46:21Z"},"agent_calls":[{"tool":"get_citation_path","arguments":{"paper_1":"1704.06888"},"arguments_in":{"paper_2":"another paper's arXiv id or title"},"call":"get_citation_path(paper_1=\"1704.06888\", paper_2=\"…\")"},{"tool":"get_concepts_for_paper","arguments":{"arxiv_id":"1704.06888"},"call":"get_concepts_for_paper(arxiv_id=\"1704.06888\")"}],"by_repo":[{"repo":"araffin/srl-zoo","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"elicassion/3dtrl","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":1,"harvested_for":["2206.11895"],"harvested_for_calls":[{"arxiv_id":"2206.11895","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"2206.11895"},"call":"get_harvested_code_for_paper(arxiv_id=\"2206.11895\")"}]},{"repo":"sermanet/tcn","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0},{"repo":"tensorflow/models","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0,"harvested_for_other_papers":4,"harvested_for":["1502.03167","1512.03385","1605.07146","1704.04861"],"harvested_for_calls":[{"arxiv_id":"1502.03167","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"1502.03167"},"call":"get_harvested_code_for_paper(arxiv_id=\"1502.03167\")"},{"arxiv_id":"1512.03385","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"1512.03385"},"call":"get_harvested_code_for_paper(arxiv_id=\"1512.03385\")"},{"arxiv_id":"1605.07146","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"1605.07146"},"call":"get_harvested_code_for_paper(arxiv_id=\"1605.07146\")"},{"arxiv_id":"1704.04861","tool":"get_harvested_code_for_paper","arguments":{"arxiv_id":"1704.04861"},"call":"get_harvested_code_for_paper(arxiv_id=\"1704.04861\")"}]},{"repo":"2023-MindSpore-1/ms-code-7","kind":"listed","samples":0,"ran":0,"instrument":0,"unverified":0}],"by_repo_harvested_for_is":"harvested_for: the arXiv ids (as Syntology's graph holds them) of every paper whose samples Syntology's graph links to this repository, in arXiv-id order, complete; the page names the first 3 and says how many more. A link only: it does not say the papers are the same work, that one copied the other, or that the code implements this paper's method; harvested_for_calls gives the get_harvested_code_for_paper call for each arXiv-shaped id among them","by_repo_is":"one row per repository in the archive's code links, Syntology's graph links or the samples; samples 0 means no sample is linked to this paper, a statement about Syntology's coverage, not about the repository: with harvested_for_other_papers present, Syntology harvested the repository for that many other papers, and without it, Syntology harvested nothing from it; placed_by_identical_code counts samples whose link records no repository, placed here because identical code was harvested from this repository for another paper; ran counts executions on a synthesized input and instrument counts failures of Syntology's instrument, not of the code"},"code_links_note":null,"arxiv_metadata":null,"syntology_extracted_results":null,"record_sha256":"4054c76ec38de130f6c8f7ee33935ceecbda7b363902ec15420a83bc99cd1a2b","record_changed_at":"2026-09-28","record_changed_at_basis":"first_hashed"}